Computing at Your Fingertips

In April at the Association for Computing Machinery’s Computer-­Human Interaction conference, the major conference for computer interface design, two research groups from the MIT Media Lab presented prototypes of input devices that—though both worn on the finger—serve very different purposes.

One device, from Pattie Maes’s Fluid Interfaces group, is a text-to-speech converter for blind users. Worn like a ring, it has a built-in camera, and it uses audio or tactile feedback to help users scan their fingers along lines of text.

The other device, a joint project between Chris Schmandt’s Living Mobile group and Joe Paradiso’s Responsive Environment group, was inspired by a cosmetic product—colorful stickers applied to fingernails, which are particularly popular in Asia. It’s a thumbnail-mounted track pad that would let users control their portable devices when their hands are full—to answer the phone or scroll through a recipe while cooking, for instance. It could also augment other input devices, allowing someone typing a text message, say, to toggle quickly between symbol sets.

The heart of the reading-assistance device is a novel algorithm that extracts visual information from text. Every time the user positions his or her finger at the start of a new line, the algorithm estimates the baseline of the text. It then tracks each word that slides past the camera. When it recognizes that a word is positioned near the center of the camera’s field of view—which reduces distortion—it isolates that word.

The baseline estimate also lets the algorithm realign the word, compensating for distortion caused by oddball camera angles, before passing it to open-source software that recognizes the characters and translates recognized words into synthesized speech.

“Anyone who needs help with reading can benefit from this,” says Roy Shilkrot, a graduate student in Maes’s group and one of the device’s designers. “We got many e-mails and requests from organizations, but also just parents of children with dyslexia, for instance.”

To build the nail-mounted device, the researchers needed to pack touch sensors, a battery, an antenna, and three separate chips—a microcontroller, a Bluetooth radio chip, and a sensor controller—into a space no larger than a thumbnail. “The hardest part was probably the antenna design,” says Artem ­Dementyev, a graduate student in Paradiso’s group. “You have to put the antenna far enough away from the chips so that it doesn’t interfere with them.”