MIT Uses XBox Kinect to Create Minority Report Interface

24,720 11 Loading
Minority Report Interface Using Kinect

Minority Report has come to life thanks to the Kinect and MIT.

In the movie Minority Report, Tom Cruise uses an advanced user interface system that lets him control media files with nothing more than a gesture from his gloved hands. He could do the same now with little more than an XBox. A member of the research staff at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) has created a Minority Report interface using the Kinect 3D sensor and a bunch of open source software. Not only does the device only cost $150, you can use it without any dorky gloves. Garratt Gallagher, the interfaces's creator, shows off some basic gesture controls in the video below. It really does look like it came straight out of the movie. Awesome.

Did Microsoft understand what they were doing when they put a $150 3D motion tracker on the market? This thing is a hacker's dream come true! We've seen some amazing projects recently that take advantage of the Kinect sensor. A student at MIT (not Gallagher) hooked a Kinect up to a robot and had it detecting people and following gesture commands in no time. Everywhere you go on YouTube you'll see new projects that use the Kinect as everything from a range finder to a puppet control system. Gallagher's Minority Report interface, however, stands out as one of the more impressive displays of the sensor's accuracy. It can detect hands from among 60,000 points of data collected by the Kinect (see the dots in the bottom half of the video), and it can track the motion of hands and fingers at 30 frames per second. That's more than fast enough for practical applications, as you can see:

Compare Gallagher's setup to the original system shown in the movie. Tom Cruise's interface may be flashier, but I'd say the two are pretty damn similar. And again, no gloves needed for Gallagher.

For those who regularly follow Singularity Hub, you'll remember that the scientist who was asked to design the original interface shown in the movie, John Underkoffler, is also from MIT. He's spent the last several years taking his concept and making it a real-world system called G-speak. His company, Oblong, is actively perfecting that platform for market. Gallagher's creation has to be a kick in the pants to Underkoffler. It may not look at nice as G-speak, but it's certainly much cheaper.

And Gallagher did it all using open source software! He relied on LibFreenect as the driver to get the Kinect talking to Linux. Point Cloud Library (PCL) handles the complex geometry created by the enormous number of spots detected by the Kinect. PCL is being developed for the Robot Operating System (ROS) by Gallagher and others at MIT. Gallagher also created his own hand detection tools for ROS which are pretty cool.

The great part of all these packages of open source software is that they can be used, modified, and shared for free. A few weeks ago, in fact, Gallagher was demonstrating a remarkably similar setup he used to create a virtual piano you could play with the Kinect. Watch it in the video below. Open source software is the path to wonderfully accelerated research...and fun musical instruments!

According to CSAIL, Gallagher wants the Minority Report interface to inspire new research with robots. His future plans for the Kinect will see it as part of a system to guide robotic manipulation. Other major robot developers, like Willow Garage, are working on similar projects. Having played with the Kinect in the way it was intended - as a video game controller for the XBox - I have to say that I like these applications much better. Hitting a virtual ping pong ball with your hand is mildly enjoyable. Controlling a real-world robot with the flick of a wrist is frikkin' awesome. Kudos, Gallagher, and keep up the good work!

[screen capture and video credit: MIT CSAIL]
[source: MIT CSAIL]

Discussion — 11 Responses

  • pk December 10, 2010 on 7:00 pm

    Considering the wild rate at which awsome kinect hacks are being created kinect and kinect derivate products may actually be that awsome product microsoft have been trying(and failing) to make for so many years.

    Now of course, being microsoft they could sink that opportunity by continuing their closed source apple-style product lock in and miss a great opportunity.

    • Anonymous pk December 10, 2010 on 7:34 pm

      At least they said they are okay with all the awesome 3rd party Kinect programs like this.
      They just don’t like hacks (of which there are none yet)

    • Medek-Medekai pk December 10, 2010 on 7:40 pm

      Absolutely agree. Its almost as if they finally created a product worth their canned “hype”, without even realizing it. As such, I still don’t think they fully realize the marketability of Kinect on the PC side of things, which is bizarre, as MS is a PC company! Wake up MS, time to do what you do best, market and capitalize!

  • Dthany December 10, 2010 on 7:47 pm

    Lucky it is good for something(hacking) because as a gaming peripheral is stinks so far.

  • Garratt Gallagher December 11, 2010 on 4:35 am

    Thank you singularity hub for a great article. I invite you all to check out my next project: putting together a cheap robotics platform bundle, so everyone can do amazing things like this, right out of the box. I’m calling it BiliBot, and you can check it out at

    • irisorn Garratt Gallagher January 11, 2011 on 7:56 pm

      Did you saw gonhoBrainsStorm hacks?? I think that you can be interested on it…

  • L.A. Louizos December 12, 2010 on 11:38 am

    All these gadgets like Kinect of Emotiv bring the resources of big companies to a low end consumer price for people with creative minds.

    I wonder why companies still insist on close source model since their gadgets will be hacked anyway.

    Open source should be the model for these companies at least for the software end….

  • Dallas Melendez December 23, 2010 on 4:24 am

    fellas, fellas, he’s just using magic and witchcraft with kinect as the medium.

    • Eallcare2 Dallas Melendez December 26, 2010 on 1:52 pm

      That exactly what I am saying. Watch out for the TV preachers. Soon, they will employ this tactics to perform “miracles.”

  • Eallcare2 December 26, 2010 on 1:48 pm

    Soon the televangelists will incorporate this in working miracles.

    • irisorn Eallcare2 January 11, 2011 on 7:54 pm

      I think that the future is robot teleoperation because we have the technology now. We must to wait 10 or more years to do thinks like minority report.
      What do you think about it??