Credibility in Cooltown
By Michael Nygard, Wed Feb 13 00:00:00 GMT 2002

Getting good advice isn't as simple as it used to be. How do you assess credibility in the modern, mobile world?


Fitting these ideas into your standard fare viewpoint article isn't as easy as it might seem. Let's try something a little different this time...

It is a warm Spring evening in Cooltown. Ray and Kimberly decide to hit Sic Transit - the new open-air cafe in their neighborhood.

Being hip, connected, and totally wireless, Ray whips out his mobile phone to check the place out. "Hey, look, it got four stars from the Port City Cafe Association!" he exclaims.

"Four stars out of how many? Four, five, or ten?" she asks.

"Uh, it doesn't say. That doesn't do much good. You keep driving, and I'll keep looking." Ray spends a little while clearing out the ads that popped up as they drove through different microzones, then finds more reviews of Sic Transit.

"Aha," he says. "FoodCritics.com has a pretty terrible review about them. Says the food was bad, but the service was so slow that he ate it anyway out of sheer hunger."

"Hmm. That doesn't sound good. Who's the reviewer?" she replies. He checks the screen again and says, "fr8daz".

"Who on earth is that?" she wonders.

"It doesn't say. I'll see if he wrote any other reviews." Ray clears a couple of ads, then says, "He wrote one other review about a place called Zibo's. Hey, it's right across the street from Sic Transit. Maybe we should go there instead?"

"Maybe he works for Zibo's," Kim says darkly. "Yeah, maybe. It doesn't say," Ray replies.

"We don't have any way to tell how credible fr8daz is," she says. "We need some way to tell how objective and knowledgable he is. You know what I'd like? A way to see if I would agree with his other reviews. There is just no way to tell how much weight I should give to some anonymous stranger's opinion."

Separating the signal from the noise

"Sure, that's been done before," Ray replies. "People have been trying a bunch of different systems to rate and assign credibility. I mean, there's Slashdot's moderation and meta-moderation. There's Advogato's trust metrics. Amazon has feedback about reviews and rates reviewers accordingly," he points out, while wondering when he started speaking in hyperlinks.

Kim snorts. "It goes back a lot farther than that, Ray. Even back when a person would live in the same village all her life, you still had to decide who to believe. You usually knew everybody in the village, so you knew who the liars were. Other than that, official news usually came from the King or the Church-always safest to assume that they were credible, right? There's a credibility function in there: I keep a vector in my head, with a value from 0 to 1 for everybody I know, plus special cases for the King and Church. Both of them get a 1."

"I suppose you adjust somebody's credibility whenever you interact with them. Add or subtract a little bit based on their behavior. That seems to work pretty well when you have a small number of connections to people that you know well," Ray muses. "I think it breaks down when you have links to too many people."

"Or when you don't know them well enough to assign a rating," she adds. "Back then, people were trained by King and Church to believe anything in writing. As a result, in the early days of printing, people would believe anything they saw in print. They had to learn to judge the source. Now imagine reading the same article in the Washington Post and the Weekly World News. Same article, but different credibility."

Ray nods, "It sounds like credibility is transitive."

"Well, is can be borrowed or lent, at least." Kimberly agrees. "I'm not convinced it's totally transitive. Any of the broadcast, multicast, or one-to-many media rely on the credibility of well-known organizations."

Ray says, "Speaking of which, Zagats gives Sic Transit a 30 out of 32. For restaurants, it doesn't get much more credible than that. Let's go ahead and give it a try."

"OK, we'll see. There's a feedback loop involved, too. If someone makes an erroneous or ridiculous statement while using borrowed credibility, it detracts from the lender's credibility."

Ray starts coughing. It sounds suspiciously like "TWA 800". Once he's done, he says, "You know, there's a word for people who can't evaluate a statement's credibility. We call them 'gullible'. Oddly enough, it's not in the dictionary."

"That's going not going to help your credibility rating."

An issue becoming ever-more important

"Huh. I didn't think it could get any worse. OK, so now, we're way beyond just print and broadcast media. We've got telephones, 'Net news, weblogs, journals, location-based messaging, email, SMS, online communities, spam, and online restaurant reviews. Now all over time and space, too? I've got to evaluate credibility for twenty-year-old Usenet messages and messages left in the air years ago?" Ray cries.

"In some special cases of small communities, like mailing lists or small websites, I may be able to keep track of a few people's credibility, but I'll never keep track of the thousands of people I intersect with! I mean, look at this!" He shows her his phone, which has five new messages for locations they've passed. "I drive back and forth to work every day. Eleven ads on my phone each way. The same twenty-two ads I delete every single day" Ray adds.

"Mechanical assistance," Kimberly replies brightly, as she pulls into Sic Transit's parking lot. "Just remember that credibility is personal. It's like a weighting factor I apply to your statements to judge them. To be effective, any system has to take that into account. It has to help me track and apply credibility ratings. Group assessments like collaborative filtering, can give me a good initial rating for any individual, but that's not the end of it. I should also be able to tell the system to automatically 'believe' individuals above a threshold, or 'reject' individuals below a threshold. Kind of like a killfile, but more precise."

They get out of the car and move to the hostess stand at the cafe. "Killfile?" Ray laughs. "Wow, are you showing your age!" He ducks. "Seriously, though, you're forgetting something. Your whole system assumes that you know who you're dealing with. You can't have credibility and anonymity at the same time."

The hostess leads them to a pleasant table for two. They sit down and pick up their menus. Kimberly says, "Let's see whether fr8daz's credibility rating is going up or down."

Michael recently joined in founding Halley's Fifth, L.L.C., a Twin Cities consulting and services firm after serving in business and technology leadership roles for more than ten years. In that time, Michael has declared himself an ally of truth, order, and clarity. He believes strongly that quality demands elegance, that doing things right can be faster, and that "later" always comes sooner than you think.