For his PhD at the University of Cambridge and then at Microsoft Research, Jamie Shotton developed a way for a computer to identify different objects in a moving video. By dividing pixels into segments according to color, the software could separate, for example, a sheep from a field, or a bookshelf from a desk.

This brought Shotton widespread attention, and one evening he received a call asking him to join a secret team working on a new video-game control system for Microsoft. The group hoped to have the system classify individual human body parts in a video stream and then allow people to interact with a game using nothing but their bodies. In the shower one day, Shotton realized that he could segment objects according to their distance from the camera rather than their color.

That led to Kinect, a motion sensor for the Xbox 360 game console that was a monumental development in computer vision and machine learning. It has not represented a sea change in computer interaction, though, perhaps because it requires too much physical effort to use one’s body in such a way for a sustained length of time. Shotton remains undeterred. Some of his hand-gesture control software will debut in HoloLens, Microsoft’s forthcoming augmented-reality device. He’s also working on allowing even basic depth-sensing webcams to interpret subtle hand movements. A user can zoom in with a simple pinch of the fingers in space, or enter a password using nothing but hand signals. “There are new and better ways of interacting with computers in the future,” he says.

—Simon Parkin

This story was updated on August 18 to clarify Shotton’s PhD work and to correct that his latest research is not yet related to HoloLens.

Watch this Innovator at EmTech 2015
Meet the Innovators Under 35