The Power of Jotting and Sketching
By Mark Frauenfelder, Mon Nov 04 12:00:00 GMT 2002

Handhelds are more powerful than ever, handwriting recognition algorithms have improved tremendously, and real solutions are finally in sight.

I've been a subscriber to the Well, a popular computer bulletin board system based in Sausalito, California, for close to fourteen years. The Well is a place where people come to discuss a variety of issues, from gardening to the Grateful Dead. For most of the Well's existence, the only way to access it was through a terminal emulator, which is a program designed to function like an ancient Teletype machine. That meant that the Well was a text-only environment. Graphics simply couldn't be displayed.

That usually wasn't a problem when you were discussing the finer points of Jerry Garcia's latest solo effort, but what about when you wanted to explain how to construct a simple irrigation system for a vegetable patch? It would have been nice to be able to include hand-drawn sketches in your posts, but in those days, that just wasn't impossible. Instead, people who needed to include illustrations in their messages had to "draw" using keyboard characters like |, \, /, and -. It was a huge hassle.

Transcribing Thoughts

Fourteen years later, you'd expect the Well to have all sorts of great drawing tools built into it. It doesn't. Neither does the rest of the online world. If you want to include a graphic in your email message or in an online discussion forum, you have to create one using a drawing program like Adobe Photoshop, and then either attach the file to the email message, or ftp the graphic to a Web server and link to it. It's not very convenient, and that's a shame, because people love to jot notes and sketch diagrams, maps, flow charts, and ideas. Jotting and sketching are great ways to communicate. And computers are a great way to transmit and store information. Naturally, people have been trying to combine the benefits of writing and sketching with the benefits of computers ever since the first digital computers were invented. But it turns out that jotting and sketching and computing haven't mixed very well.

Sure, there are plenty of wonderful drawing programs, but they're primarily designed to produce art that is more permanent than a quick sketch of a map to a campsite or a dinner table arrangement. And handwriting recognition is even more problematic. Who wouldn't love to have a stylus and a small tablet to jot and sketch on, and be able to instantly share the pages with other people wirelessly? And if the handwriting could be saved as searchable text, that would be even more useful.

Character Recognition

But the accuracy of handwriting recognition is notoriously lousy, unless you use a constrained character input system like Palm's Graffiti, which forces the user to write characters in a certain way to make each letter distinct from one another.

One person who knows about the perils of handwriting recognition all too well is Dr. Leonid Kitainik, who has been developing handwriting recognition and electronic ink technologies for 23 years. Kitainik was the leader of the team that developed the handwriting recognition for the Newton MessagePad, Apple's personal digital assistant that became famous in 1993 as an example of how a $500 gadget loaded with the latest technology was less useful than a fifty-cent scratchpad. Today, Kitainik writes off the Newton's failure as a result of Apple being "too visionary." He says, "The Newton's hardware and system could not support the ideas, and a lot of mistakes were made."

But times have changed, says Kitainik, who is now the general manager of ParaScript, an offshoot of ParaGraph, the company that developed the handwriting recognition technology for the Newton. Handhelds are much more powerful than the Newton, and handwriting recognition algorithms have improved tremendously since 1993.

ParaGraph's software has worked its way into a wide variety of platforms. After Newton, ParaGraph went on to develop a program called CalliGrapher, which converts stylus handwriting into text using a combination of fuzzy logic and neural net techniques. ParaGraph was later sold to SGI, and was sold again to Vadem, which sold the technology behind CalliGrapher to Microsoft. Today, Microsoft uses CalliGrapher's technology in the handwriting feature found in its Tablet PCs and Pocket PCs. (Microsoft calls it Transcriber.) The latest version of the Macintosh operating system, OS X 10.2 (aka Jaguar) also offers a handwriting recognition system, Inkwell, which has its roots in ParaGraph's technology.


The latest product to come out of ParaScript's labs doesn't have anything to do with handwriting recognition (not yet, anyway). It's called riteMail, and it's an application that allows desktop, Palm and Pocket PC users to email handwritten notes and drawings to each other. Basically, it's the thing I've been waiting for since 1988. Kitainik demonstrated riteMail by sending me an email message.

When I opened the email, I saw his drawing of a funny face, along with a handwritten greeting. To reply, I clicked on a link in the message and a Java application opened in my web browser. RiteMail's interface is very simple, and in no time I had drawn a mustache on the cartoon face and returned the message to Kitainik. The handwritten notes were not converted to text, but remained just as they had been written, in a variety of colors and line widths.

The beauty of riteMail, says Kitainik, is that it frees mobile users from fax machines and high-end drawing applications. He gives the example of an executive on the go who has a wireless PDA, and needs to quickly draw and email a new organizational chart to her assistant. If she's at an airport or hotel, she'll have to hunt down a fax machine or use the business center's computer to draw the org chart. But if she has the riteMail app installed on her wireless device she can draw a freehand chart using her stylus right on the touch screen.

riteMail has a nifty shape-recognition feature, called riteShape, which can sense when you are drawing a circle, square, or triangle, and automatically adjust the angles and lines so that the shapes end up looking perfect. Then, the executive can email the org chart to her assistant, who can either redraw the chart using PowerPoint or print it out as is.

Conversion Quandaries

As it stands, riteMail is a nice application. But to be truly useful, it will have to gain the ability to convert handwritten notes into machine-readable text, because otherwise the notes won't be searchable. Kitainik knows this, and ParaScript is working on it. Incorporating handwriting recognition technology into riteMail won't be easy, however. Unlike Palm's Graffiti, which manages to do a decent job of converting handwriting into text by constraining users into entering special pen strokes, one character at a time, riteMail will have to figure out how to convert "natural handwriting recognition" - that is, unconstrained print and cursive handwriting - into electronic text. Even though the processing power and memory of computing devices has boosted considerably since the advent of the Newton, natural handwriting recognition is a major challenge. People often write "a" and "u," and "f" and "t" exactly the same way, for example. Different nationalities use different kinds of shorthand. For instance, German writers typically squash the last few letters of a word together, "swallowing the end of the word," says Kitainik.

As if these issues weren't hard enough to deal with, Kitainik says that "truly handwriting recognition shouldn't require training. People want good results right of the shelf, or they will abandon it. We're not big believers in pre-training. It's a challenge for us."

Not one to shirk from a challenge, Kitainik believes the way to tackle the problem is by using word-recognition rather than character-recognition, because that's the way people read. ParaScript is developing a handwriting recognition technology, called riteScript, which uses whole word recognition technology. I tried the demo of the work-in-progress version of riteScript on ParaScript's site and was impressed with the results, especially when I recalled how abysmally the Newton was at converting my scrawls into meaningful text.

The reason riteScript performs so much better than the Newton, is because it uses a different technology. While both the Newton and riteScript analyze written input as whole words, riteScript's system relies heavily on context -- that is, it looks at pairs of words and consults a database to determine the statistical likelihood that one word follows another. It also uses more specialized tricks to improve accuracy. For example, if the software sees a word with an "@" and a "." in it, it'll assume it's an email address and match the domain name with a database of common domain names.

ParaScript is big on the idea of using a "natural pen interface," an input method that reverses the role that most document-producing applications take. Take PowerPoint for instance. It offers a wide variety of tools and menu items. But ParaScript's interface offers just a few tools, and relies on algorithms to figure out how to convert a user's input into text, drawings, charts, demarcation lines, arrows, and bullets.

Kitainik's ultimate goal is to combine electronic ink, shape recognition, and word recognition technologies into a "full notes recognition" platform, which not only does a good job of performing each function independently, but also makes everything work together (so it doesn't make mistakes like confusing a circle in a flowchart with the letter "O"). The result will be a system that interprets what the user is jotting and sketching, and turns it into a well-formatted, machine-readable document, much like a PDF or RTF file. "That's the new category we are building," he says.

I'd say it's about time.

Mark Frauenfelder is a writer and illustrator from Los Angeles. (And recently starred in an Apple Switch ad!)