Mistry's infrared system makes a mouse obsolete. Just use your hands.

Take your hand off your computer mouse, Pranav Mistry from MIT is ready to make it disappear. He’s designed ‘mouseless‘ – an infrared system that tracks your hand motion and translates it to cursor movement on your screen. By simply holding your hand in a curved position you can point and click, right-click, scroll, and drag, with nothing but open air. The cost of the system (in parts) is just $20. This is another very cool user interface solution from Mistry, who has earned widespread acclaim for his Sixth Sense personal augmented reality system. Check out a cutesy demo for mouseless in the video below. The real action begins around 1:02.

Most user interface designers seem to agree that the mouse and keyboard arrangement that’s been popular so far can only last so long. No one knows, however, what exactly will replace it. Gesture technology is leading the pack, with video game and TV applications coming out this year. There’s a certain appeal to waving your hands in the air and controlling your machines like some sort of conductor of a robotic orchestra. The more versatile of these systems are dependent on gloves for the moment, but they’ll likely drop that restriction in the next few years. Mistry’s mouseless project is already a step in that direction, letting you use nothing but your hand to control your computer.

The ease in which you can use the mouseless setup belies it’s rather ingenuous system of hardware and software. A small infrared laser is fitted with a line cap so that it projects a thin plane of illuminating IR light near the flat surface it’s resting on. Your fingers catch and reflect this light and give off an IR glow. An infrared camera picks up the glowing points of light and computer vision software translates their location and configuration into commands for the cursor. The computer can distinguish between general movements, tapping with index or middle finger (which it translates as left and right clicks), flexing of all four fingers (used as a scroll feature), and a pinching motion used to drag and drop.

The defined gestures for 'mouseless' give it much the same capabilities as your average computer mouse.

There are a couple of things that I think this user interface really gets right. First, it lets you rest your hands on the table top. Proponents of gesture controls often seem to forget that waving your arms in the air can get very tiring, and precision starts to fail as your shoulder aches. Mistry’s invisible mouse has much the same benefits of gesture controls but anchored to a flat surface you can rest on. Second, mouseless gives you a very primitive sense of physical feedback. While you can’t feel the IR laser, you can feel the table you’re sliding and tapping against, giving you some idea of if your motions are broad enough to be detected. This is far from the advanced haptics that we are likely to see in user interfaces of the far future, but it’s still better than nothing at all, which is what most other gesture controls offer.

Of course, there are some limitations to the invisible mouse as well. Foremost among these is the fact that it’s still trying to be a mouse. The mouseless project is pretty narrow. It’s a baby step forward in user interfaces when we may really need a leap. It also is limited to two dimensions, when we think and move in three. Keeping itself in 2D is probably why the project can be so easy to use and cheap, but it limits the long-term advantage over current hardwired systems.

Despite it’s cool appearance, the mouseless project is really just a modified version of previous mice devices. We keep trying to build a better computer mouse and that’s both pragmatic and frustrating. This invisible mouse is neat, but it doesn’t really offer much more than an air mouse (which I still hate as a concept). I’m sure that Mistry and others will find some good uses for a mouse-like gesture control system that costs just $20. Yet the real story is likely to be the future devices that Mistry builds using the IR laser and camera technology. That tech might be able to make the controls of Sixth Sense more robust, it could convert normal monitors into touch screens, or it could make any surface interactive. There’s a lot of potential there, much more than can be adequately contained in a mouse.

[image and video credits: Pranav Mistry]
[source: PranavMistry.com]

Skinput uses pico projectors and acoustic monitors to turn your body into an I/O for mobile devices. Sounds good.

Electronic devices are getting smaller, and so are their interfaces. If you’ve ever had problems typing on your mobile, or changing a song on your iPod while jogging, Chris Harrison has the answer. His Skinput prototype is a system that monitors acoustic signals on your arm to translate gestures and taps into input commands. Just by touching different points on your limb you can tell your portable device to change volume, answer a call, or turn itself off. Even better, Harrison can couple Skinput with a pico projector so that you can see a graphic interface on your arm and use the acoustic signals to control it. The project is set to be presented at this year’s SIGCHI conference in April, but you can check it out now in the video below. Skip past 1:00 to avoid the intro, but make sure to catch the Tetris game demo at 1:19, and the cool pico projector interface starting at 2:05.

Incorporating your body into your mobile systems could be the next big theme in human computer interfaces. The Skinput system bears a strong resemblance to the I/O for Sixth Sense, Pranav Mistry’s open source personal augmented reality device. Whereas Mistry uses a camera to capture video input, Harrison has focused on acoustics. Researchers at Microsoft have taken a third tactic: using the electric signals on the skin’s surface that correspond to muscle movement as input commands. All three are pretty novel approaches to controlling your technology through your body, and any one might make headway in the next few years. Together, they suggest that the best way to take advantage of miniaturized mobile devices is to make them part of you.

I’ve been impressed with Harrison’s work at Carnegie Mellon’s Human Computer Interaction Institute since I reviewed his Scratch Input device last year. That system used acoustic signals on hard surfaces to control various electronics. Skinput is the logical successor to Sratch Input, taking the basic premise of an acoustic interface and upgrading it with projected video and augmenting it by placing it on the body.

The actual acoustic monitor for Skinput is pretty large. Of course, it's still a prototype. If an acoustic I/O ever came to market, it'd have fit inside a standard iPod armband holder.

As advanced as Skinput may be, however, it’s hard to know if it will ever make it out of the prototype phase. Novel I/O systems are amazing to watch, but transitioning them into a marketable product can take years. Here’s an idea though: Mistry’s Sixth Sense device is open source, maybe Harrison should try to combine it with Skinput. Together, visual and audio input could give the personal AR system a dynamic and sensitive set of complex controls able to take on any task.

Whenever I see this level of engineering and inventiveness I’m as impressed by the people behind it as I am with the technology itself. Harrison’s already on my radar after Sratch Input, even more so now with Skinput. Prototypes like these help shape the dialogue about the future of human computer interfaces even if they never make it to market. (Relatively) young inventors like Harrison, Mistry, and countless others are going to make this century a truly remarkable one to live in. Maybe in a few years I’ll be writing these posts on my arm while I’m out jogging. Make it happen, Harrison!

[image and video credits: Chris Harrison, CMU HCII]
[source: Chris Harrison website]

At TEDIndia Pranav Mistry announced that his SixthSense system would be open source.

Have you ever looked at an object in a display window and wondered how much it cost? Or have you seen someone you’ve met before but can’t remember their name? It’s time you got MIT’s version of the sixth sense. The creation of Pranav Mistry, the SixthSense system combines a pocket sized projector, a mirror, cell phone, and a camera to create an augmented reality experience you can carry around on your person. We explored some of the amazing capabilities of SixthSense back in February and things have only gotten better since then. At the recent TEDIndia conference, Mistry announced that he would be making SixthSense available through an open source license. That’s right, you’ll be able to build your own device and upload the software for free. How cool is this thing? Just check out the standing ovation that Mistry and his MIT advisor Patty Maes received at an earlier TED conference in the video after the break.

Patched together from off the shelf components, Mistry has stated that the SixthSense system will likely cost less than $350. To facilitate the proposed open source licensing of the device, Mistry will provide instructions in how to assemble the hardware as well as provide the free software to run the components in tandem. That’s great news to everyone interested in the device, and better news to everyone who believes in the open source and open hardware movements. As more developers demonstrate that open licenses promote product improvement and create a more competitive device, we could see more innovators willing to try it. So thanks to Mistry for creating an awesome piece of technology and even bigger thanks for having the foresight to let that technology be accessible to everyone.

[photo credit: TEDIndia]
[video credit: TED 2009]