Microsoft's Magic Pen

If Jian Wang had his way, everything would be digital.

“I hate printers-they turn digital things into analog,” he jokes, wading through a sea of cubicles at Microsoft Research Asia in Beijing, China. Fortyish and lanky, the computer scientist specializes in inventing new computer interfaces to bridge the gap between analog and digital. His own interface, though, is a wide smile, which complements his denim shirt and easygoing manner.

Stopping at a desk, Wang picks up a rectangular, silvery pen about the size of a magic marker and scribbles some corrections on a paper document. But this is no ordinary pen. A few seconds later, his comments appear on a nearby computer screen-superimposed on the electronic version of the document in the exact spot where he wrote on the hard copy. Wang’s pen captures handwriting and lets users make changes to digital files-on paper.

This “universal pen,” as Wang calls it, could transform the way people interact with computers. Unlike gizmos that write on computer displays or special pads of paper, Wang’s invention uses regular ink, works with regular paper, and lets users combine handwritten text and diagrams with digital content from reports, magazines, and Web pages. An executive on a plane trip, for instance, could mark up a paper copy of a report and later transfer the changes to the file on his or her computer automatically.

Wang’s digital pen also reflects an ongoing transformation in the process of invention at some large corporate labs, a hybridization of the lone inventor and traditional corporate R&D. Wang is the pen’s lead inventor, and it is his insight, daring, and creativity that have largely driven the effort to develop it. But at the same time, he could not have made such rapid progress without Microsoft’s collective expertise in pattern recognition algorithms, computer vision, handwriting technologies, and text-editing software. “Personally, I’m really excited about it,” says Rick Rashid, senior vice president of Microsoft Research, whose main facility is in Redmond, WA. “It’s an example of a new kind of product incubation that we do,…one that brings together people with many different skills to solve a unique problem.”

Microsoft Calling

Himalayan stacks of papers fill Wang’s office, despite his proclaimed distaste for printouts. Amidst these reminders of the analog world, the inventor describes the beginnings of the pen project. As a professor of engineering psychology at Zhejiang University in Hangzhou, China, Wang had made his name in human-computer interfaces and virtual-reality systems. Then, in 1998, at the height of his academic career, he received a cryptic e-mail from Kai-Fu Lee, a renowned researcher who was just starting up Microsoft’s Beijing lab. Lee suggested they meet but was too busy to explain why. “I don’t know how he got my name,” laughs Wang, “and he didn’t explain who he was.”

Feeling adventurous, Wang went anyway. He quickly discovered that he and Lee shared the ambition to create a user interface for computers that was based on handwriting. The pen project was inspired in part by Wang’s desire to enable mobile computers to handle handwritten Asian languages. But from his academic research on 3-D interfaces, he had learned that if technology isn’t designed to be practical and appealing to a variety of users, it won’t be widely adopted. “We wanted to make a breakthrough, not improvements,” he says, adding that in Microsoft he had found a perfect partner. “I realized this is the place to be if you want your invention to be used by millions of people instead of just a couple.” With that motivation, Wang left Zhejiang University and joined the Beijing lab in the fall of 1999.

Immediately, Wang, Lee, and a core group of researchers began daily brainstorming sessions to iron out the project’s goals-sessions that eventually included their top boss. “When we talked with Bill [Gates] about this technology,” says Wang, “we realized we are inventing a new kind of document, not just a pen.” The key idea, which fits with Microsoft’s business of making office computers more useful, was that a document could be kept digital even when printed-with the right kind of pen interface and software.

It took the better part of four years to make it work. Not only did Microsoft stick with Wang, but it also allowed him to build a team of about 20 researchers, giving him access to the skills in programming and hardware he needed to design the pen. After a few false starts-including a cumbersome version that inferred what the user was writing from the pen’s movements-Wang’s team took advantage of advances in computer vision algorithms and based the pen’s sensing on a simple digital camera.

The researchers’ first challenge was finding a way to determine the pen’s position on the page. Their solution involves special software that puts a barely visible background pattern, like a watermark, on standard copy paper as a document is printed (see “Putting Pen to…Computer,” below). That enables the computer to figure out not only exactly where the pen is relative to the document but also which document is being modified, because each page has a unique code. A pressure sensor in the pen triggers a tiny embedded camera, which snaps pictures of the user’s writing. The images are stored in the pen on a memory chip like those found in digital cameras; when brought within a few meters of a PC or laptop that has the proper software installed, the pen transmits the images wirelessly using a Bluetooth connection.

Putting pen to…computer

The pen contains a digital camera, a pressure sensor, a Bluetooth radio, and a memory chip. As the user writes on paper, the camera snaps tiny images of the ink.
The paper has a background pattern, like a watermark, that is printed with the document. The computer uses this code to figure out exactly where the ink is on the page.
The computer receives the sequence of images wirelessly and reconstructs the handwriting in the right place in the digital document file. With special editing software, the user can then manipulate and format the handwriting.

Interpreting those images and incorporating them into digital files turned out to be a tougher problem. The trick, Wang explains, is getting the computer to recognize different types of writing and drawings-to know what’s a box, what’s a sentence, and what’s a doodle-just from a series of photos. First, computer vision algorithms classify sequences of marks as words, diagrams, or shapes, all of which can be manipulated. Then character recognition software-the subject of years of intensive research at Microsoft and elsewhere-makes sense of the handwritten text. On the computer screen, the user’s marks show up as handwriting embedded in the document. Software tools can then convert the writing into typed text and rendered graphics; with these tools, the user can manipulate, say, boxes and text from a hand-drawn flow chart.

In addition to enabling users to import and manipulate handwritten text, tables, and charts, says Wang, the pen will allow multiple collaborators to make comments on separate printouts of a document; the computer could then integrate them all into the same file. The end result, says Wang, will be an interface like a portable scanner-but one smart enough to “understand” the images it captures and to fulfill its inventor’s dream of turning mounds of “analog” paper back into digital files.

Away from the lab, over a meal of Shanghai-style meatball soup and steamed fish, Wang explains how the lab’s culture promotes invention. Besides sharing a “work hard, eat hard” mentality, he says, the lab’s members-and leaders-are experienced researchers who understand that breakthroughs take time. “You don’t always have to prove yourself in a month, or even a year,” says Wang. So researchers have the flexibility to try a variety of approaches-important for ambitious projects like reinventing the pen.

Microsoft Research Asia is a large corporate lab, employing 150 full-time researchers, but it has surprisingly little bureaucracy, Wang says. No special approval procedures. No heavy-handed mandates from above. The company attempts to nurture invention by giving its engineers the freedom to explore interesting avenues of long-term research, while staying focused on near-term results by setting milestones and conducting project reviews a few times a year. That makes it “more like an academic-style setting,” says Wang. “We publish papers, go to conferences, and host many visiting professors. It’s an open environment.”

Observers say that this openness is essential to the lab’s productivity, especially in a country with a limited history of corporate research. “They’ve been a success because of their ties to academia,” says Shiqiang Yang, executive vice chair of the Department of Computer Science and Technology at Tsinghua University, one of Beijing’s leading engineering schools. The free exchange of ideas with students and professors has strengthened the lab’s standing in the academic community-and that helps it recruit some of China’s top students and computer scientists.

The close ties are paying off: more than 70 technologies developed at the Beijing lab have found their way into Microsoft products in areas such as video game graphics and speech recognition for dictation. As for Wang’s digital pen, it could open up whole new markets for Microsoft. Although commercialization is still a few years away, Wang’s team is in discussions with the company’s product development groups in Redmond. Specific product plans have not been made yet, says Rashid, “but it generates a lot of exciting ideas about what might be possible.”

In fact, technologies developed by Wang’s group in the course of the pen project are already paying dividends to the company. Software that can recognize and manipulate handwriting on a screen is a feature of the Tablet PC operating system Microsoft released in 2002. The Asian market could be the first point of entry for interfaces like the digital pen because of the demand for software to recognize and edit Asian-language characters, the sheer number of which makes them tedious to input using keyboards.

With any new invention, there are numerous challenges in transforming an exciting prototype into an actual product; but here, Wang’s training in psychology and human factors should serve him well. The pen and its software need to be made easier to use, he says; customers won’t tolerate added complexity. With the remaining kinks worked out, however, the invention has the potential to become ubiquitous, because of the familiarity and convenience of ordinary pens. “The pen is so pervasive, so expressive, and one of the best inventions,” Wang says. “So I think the pen could be a very good computing device for the future.”

Ultimately, says Wang, everything on your desk-books, journals, printouts-will have a digital connection to your computer. It’s what he calls “closing the loop between analog and digital.” And with increases in computing power and memory, a device like Wang’s pen could become the main computer interface people use on the road. Wang envisions that users could not only enter and store data but also, with a wireless handheld computer, access information in, say, a magazine article just by underlining words that can be recognized and “looked up” on the Web.

With so much to do, though, Wang has no time to waste. He might work for one of the world’s largest corporations, but he still keeps the long, nighttime hours of a lone inventor. After dinner, while most of Beijing is dark and quiet, he goes back to work, intent on penning the future of human-computer interfaces.