Annotating the Planet, Microsoft Style
By Howard Rheingold, Thu Oct 07 11:00:00 EEST 2004

I wasn't surprised when Marc Smith combined a barcode reader with a handheld wireless computer and fed the barcode info to Google. He had been talking for years about "annotating the planet."

A sociologist for Microsoft Research, Smith has long been interested in what would happen socially if people could attach informational annotations to objects and places. The project that grew out of his long-standing interest, the Advanced User Resource Annotation System (A.U.R.A.), provides "the ability to access and author annotations on objects and places using machine readable tags. In our system, a user can associate text, threaded conversations, audio, images, video or other data with specific tags. Users can also review the tags and descriptions of the objects they have encountered and annotated in a custom web portal. Users may selectively make the items they have scanned public to other AURA users resulting in a collectively authored database rating, reviewing and commenting on a wide range of objects, products and places. Physical annotations can be shared with other users and selected by users’ reputation statistics and other properties."

I remember the night Smith showed me his first rough prototype. I was in his living room in Redmond. I hefted the iPaq he handed me and said: "It looks interesting, but what do I do with it?"

"Just point and click," he replied: "look for barcodes, point the iPaq's camera at them, and click."

So I headed for his kitchen and scanned the first thing I saw, a box of prunes. Reading the barcode and sending a query to the Universal Product Code database returned a number of informational fields provided by the manufacturer. One of those fields revealed that the product was distributed by "Sun-Diamond Grower's Cooperative." The name conjured an image of organic hippies in California, but when I clicked on the Google link, the top half dozen hits included two especially juicy sites. One was about "U.S. v Sun-Diamond," a US Supreme Court ruling regarding the $700 million/year consortium's lobbying practices, and another headline from a political action site called that proclaimed "methyl bromide barons subvert democratic process."

I began to understand what Smith meant when he said, "Every object has a story to tell," after he handed me his homebrew scanner. This is not exactly the kind of information you normally get from a food label.

Smith pointed me toward another example in his kitchen that he had discovered himself when he first started clicking around – Kellogg's Cracklin' Oat Bran. One of the top links for that breakfast cereal revealed that a mislabeled early batch had failed to disclose dairy and egg content, which could have been fatal to anyone with severe allergies.

Although both of those early experiments in Marc's kitchen pointed to possibly enormous potential for consumer collective action, that's far from the only potential use for such a device. While obtaining information about products is one possible use for mobile barcode or RFID scanners, the social possibilities of a taggable world is what attracted Smith -- the ability to share information and to use reputation and recommendation engines to inform your decisions as you navigate the physical world. If you click on a book, you could find out not only what the major media reviewers had to say and what your neighborhood or online book club had to say, you could also add your own opinion. Or click another button, and the book will arrive at your house within a few days. If you and your buddies exchange data on the media and objects that interest you, a recommendation engine similar to the one used by Amazon could tell you that what other books, records, movies, fashions or web sites might also interest you.

Nothing about this technology is beyond the state-of-the-art. Indeed, one purpose of Smith's first iPaq version was to show that a world-tagging device could be built from off-the-shelf parts. The hardware and software to make this possible was only a necessary start to the sociologist in Smith. He has always been interest in what such technology could enable people to do things together socially, in ways they weren't able to do before.

I wonder how many of the objects, from books to chainsaws, really need to be carted home from the store if pointing and clicking could deliver them to the door. Retail outlets might no longer serve as distribution centers, serving as showrooms -- and perhaps social spaces of some kind -- instead. "The widespread distribution of such devices is likely to have dislocating effects in many sectors of life. Retail environments seem the most likely to change as consumers bring the power of the Internet to bear at the point of sale."

The title of the Aura web site is "annotate the planet." Smith gave me a guest account to poke around there. A few Microsoft Research employees are walking around today with PDAs and phones that can scan packages, search and annotate the scans, and post annotations to a website. Logging into the site, I can see what others or I have recently scanned or annotated, read annotations and search MSN,, eBay, Google, and Google Groups for the scanned item. I could add my own annotations, if I had an opinion to share about the item.

Keep in mind that devices that connect you to the infosphere also connect the infosphere to you. State agencies and your nosy in-laws will know your every move, and if you are a target of one of those with access to the surveillance sphere, you won't find it easy to elude them. Mostly, they will use their powers and their communication channels to give you 20% off on your favorite detergent. The ordinary everyday transaction, however, when it is multiplied by ALL the transactions taking place in the world, is a mighty force. If billions of people begin to use purchasing power to support their values and beliefs, what kind of economics, what kind of governance will result? When we routinely connect our telephones to the physical world by reading or tagging, will we become more or less free than we are today?

And can we have a conversation about that question before mass-adoption makes it moot?