MIT Engineer Creates 'Minority Report' Kinect

Thanks to an MIT engineer, some of the technology in The Minority Report, a blockbuster science fiction movie starring Tom Cruise, is available today.

Garratt Gallagher, a systems robotics engineer at the Massachusetts Institute of Technology, has figured out how to hack into the Microsoft Kinect and locate the hand and figure motion sensor embedded within the gaming console add-on. Once he figured that out, he created a graphic user interface (GUI) similar to the one shown in The Minority Report as a demonstration of the hand motion. In his demo, he was able to control and shift around digital images on the interface and by simply moving his fingers in the air.

The Kinect gives out clouds of 3D points at 30 frames per second, and each frame contains around 60,000 points in the raw format. What I did was use specialized 3D search algorithms to locate the hands, and segment the fingers. The basic output of that program could be seen in the lower half of the screen. To demonstrate how useful it is to track hands and fingers, I made the GUI. It included some basic gesture recognition, so I could swipe, make a circle, and frame on abject to interact with the pictures, Gallagher said in an email.

He said the hard part of the experiment was tracking the fingers and hands. With that completed, he said it would be easy for engineers to create various interfaces on the Kinect. To demonstrate these capabilities, he created a piano interface that played the instrument by him simply playing the air with his fingers. He said interacting with objects in 3D, such as assembling 3D models of an object, is one possibility as an interface on the Kinect.

Willow Garage,a company which has created an open-sourced robotics platform, provided the architecture which Gallagher based his code on. He said without them, it would have been impossible to locate the motion sensor in the Kinect.

A system such as ROS really helps researcher really maximize our potential, Gallagher said. The architechture from Willow Garage allowed him to create the hand detection program without worrying about little details such as message passing and low-level driver interfaces. Moving forward, using his findings from the Kinect, he has decided create an open-source robotics platform for the Kinect at www.bilibot.com.