Touchscreens and swarm robotics go together like geeks and video games.

Mark Micire’s PhD dissertation puts robotic control at his fingertips. The UMass Lowell student developed a command and control program for the Microsoft Surface touchscreen so that swarm robots can be easily guided. Watching Micire’s program in action makes it look like he’s playing StarCraft, only with real robots. Teams of bots can be color coded, groups can be selected by circling them with a finger, and robots can be commanded to move either individually or en masse. You can even manually drive a robot with a special pop-up interface. What’s the use for this multitouch control system? There are military applications, but Micire has a strong background in search and rescue robotics. Touchscreen swarms could be the next innovation in disaster relief. Watch Micire’s demo his system in the video below.

Swarm robots come in a huge variety of shapes, sizes, and capabilities, but they generally follow one strategy: many hands make light work. Bots are used together to divide and conquer a problem, overcoming challenges with a large number of workers. Many of the swarms we’ve seen are autonomous. In critical missions, however, robot AI may not yet be sufficient to find a solution to a problem. That’s why human guidance is still very important. Micire’s touchscreen program gives the operator various levels of control. This would allow humans to adjust their involvement as the case warrants. If robot autonomy can handle a situation, commands could simply instruct bots in a general way – go there, look for human bodies, report back. The system also allows for more direct commands – follow this route, etc. When direct human control is needed, operators can manually drive bots using the DREAM interface (Dynamically Resizing Ergonomic And Multitouch), as you can see at 2:23 in the video below.

Unfortunately, the demo above doesn’t show the system controlling physical robots. The bots on the screen are simulated. But Micire can control real world robots using the touchscreen, as you can see in the video below:

Micire’s work seem immediately applicable to fieldwork for robotics. It’s really no surprise considering his experience in search and rescue operations. It’s important to remember that robots already form a valuable part of S&R teams, allowing humans to explore areas that are too hazardous or difficult to reach. Case in point, here’s Micire’s video of work he did in Mississippi after Hurrican Katrina back in 2005

Of course, there are many applications for guided swarms besides disaster relief. Autonomous and guided drones are valuable assets in modern warfare, and advanced control systems like Micire’s could improve their usability. We’ve already seen how a similar technology (telestrators) are being developed by the US Air Force. Additionally, touchscreen controls could help swarms tackle industrial maintenance, exploration, or even surgery.

Swarm robotics holds a lot of potential, but most of the focus we’ve seen has been on the robots themselves. Micire’s work shows an interesting way in which humans can be readily inserted into the robot’s decision making process. In the near term that will allow these swarms to perform better as human intelligence still exceeds AI. Once autonomy out-paces human decision making we may still use such command systems as a means of providing over-arching control of our robots. It will be interesting to see if the potential of Micire’s program attracts any real-world applications in the years ahead.

…Maybe someone needs help fending off a zergling rush?

Video of Mark Micire’s complete PhD defense can be found here.
[screen capture and video credits: Mark Micire]
[source: UMass Lowell, Mark Micire]

Tan Le let a first time user demo how the EPOC headset translates brain activity into commands.

Tan Le isn’t just interested in reading your mind, she wants to have fun while doing it. Le is one of the founders of Emotiv, maker of the EPOC headset – a streamlined EEG device that translates your thoughts and facial expressions into computer inputs. It’s been hyped as bringing a new dimension to video games. At this year’s TED conference, Le gave a wonderful presentation on EPOC, letting a first time user demonstrate how it could control a simulated object. Available as low as $300, EPOC is one of the least expensive brain computer interfaces you can find on the market. Watch the Emotiv headset in action during Le’s TED Talk in the video below. She reveals parts of the UI and capabilities of the device that you haven’t seen anywhere else.

Calling the EPOC a brain computer interface is a little bit generous. With only 14 channels, it’s a fairly simple version of EEG. Certainly we’ve seen more complex BCIs that are currently on the market. Yet EPOC is an order of magnitude cheaper than these more advanced devices, and seems to be able to translate a wide variety of brain activity into computer commands. In the TED demonstration we are able to see a first time user pull a virtual object (~4:55) and even make that object disappear (~6:45) using monitored brain activity. In both these cases EPOC only had a single point of reference from which to base the commands. Clearly the learning algorithm of the system is fairly capable. Eventually, commercial BCIs like EPOC may enjoy a wide range of applications. Le highlights a few in the video below, including video games (8:20), automated machines (8:50), and wheelchairs (9:30).

As we discussed in our original review of Emotiv, the EPOC headset may not make it as a video game controller. Sure, being able to lift an object with the power of your mind sounds cool, but the reality looks difficult and unwieldy. I’m not sure I would spend $300 on it. Sharing my facial expressions through online avatars is only somewhat appealing, and having a virtual landscape that adjusts to my emotions seems like a recipe for disaster. Also, at the $300 level you can only use the headset with approved software applications, which are very limited at this point. Bottom line, I’m much more impressed that the EPOC system can perform all these video game tasks than I am with the game applications themselves.

Maybe Emotiv agrees? They’ve made their SDK for the EPOC very affordable – starting at just $500. Research packages are a little more expensive ($750) and advanced enterprise and education adopters will likely pay $2500-$7500 for development. These are all relatively cheap prices, and may encourage third party developers to find that killer app that the EPOC needs.

I have really no idea what that app will be. Other commercial EEG devices are either typing tools for helping patients with locked-in syndrome communicate (high-end), or novelty products for hobbyists (low-end). EPOC is clearly not the latter, but I don’t know if it has the spatial/temporal resolution for the former. I doubt it, otherwise I think Le would have shown us that at some point. Still, with a reasonable purchase price, and a robust system of capabilities, chances are that EPOC could find a good use fairly soon. It’s a great example of how BCIs are getting cheaper and more powerful as companies explore the technology. One day brain-computer interactions will become so sophisticated that traditional human-computer interfaces, like the keyboard, will become obsolete. We’re not there yet, but Le’s presentation gives me hope that we’re on the right path. In preparation for the coming BCI revolution I’m doing a bunch of mental exercises. Picturing my computer disappear is strangely relaxing.

[screen capture and video credit: TED Talks]
[source: Emotiv]

Displax's thin polymer film can be applied onto any nonconductive surface to create a touchscreen.

Portuguese engineers have taken us one step closer to fusing the digital and physical worlds. Displax, a fledgling tech company in Braga, has developed a device that can read the electrical disturbances in a thin transparent polymer film embedded with nanowires. Any pressure on the film (from a finger, or even from wind) can be detected and understood as a command. Apply the Displax Multitouch Technology (DMT) to any noncoductive surface and you’ve got an instant multitouch interface. According to their press kit, the DMT can interpret up to 16 fingers on a 50 inch screen. The film system can work on any surface from ~7 inches (18cm) to ~10 feet (3m). With drivers for Windows, Mac OS, and Linux, the DMT should be able to work with most conventional computers. Which means that all those non-touch LCD screens out there could be easily updated to be touch capable. With projectors every glass window, every wooden bench, every plastic surface could be used as an I/O device. The possibilities are awesome. Displax put its technology on display in several case studies. We have video of a concept store and a museum for you to enjoy below. Behold the power of film.

Touch surfaces are emerging as a dominant contender for the next human-computer interface. Many smart phones are now touch capable (more’s the pity for my meaty fingertips) and the iPad is likely to popularize the same concept for tablet computers (though Apple isn’t the first to merge those technologies). We’ve also seen touchscreens make impressive splashes in novel settings like the Ring Wall in Germany and the Hard Rock Cafe in Las Vegas. These things are practically everywhere. Now, with Displax, that may take on a literal truth. Most of the surfaces around us are nonconductive and less than 3m in size. The DMT could go on almost everything. What’s the world going to be like when anything we touch could interpret our actions as commands?

The Displax Multitouch Technology has some impressive specs. It’s completely transparent (to the naked eye), a 50 inch screen with processor weights just 300 grams, isn’t affected by lighting conditions, and it’s sensitive enough to detect you blowing on it. Curved surfaces are ok, and it works indoors and outdoors. 16 fingers on a single 50 inch surfaces is probably more than enough for most applications, but Displax is hoping to increase that limit later. The DMT will start shipping in July 2010 with prices to be announced later. It will come with a software package that includes applications for photos, videos, and Google Maps. All in all, it’s a tidy little package.

We've seen these types of systems a thousand times before, but the crayon wall at the Ecomuseum of Barroso is still my favorite application from the videos.

But it’s only half complete. Displax provides the touch interfaces, but they don’t provide the display. Maybe that’s a quibble, but it’s going to affect how this technology is applied. Put the film on a screen and it becomes a touchscreen, with flashing icons and things to indicate your commands are being interpreted. The same holds true for a surface with an image projected onto it. However, for a non-screen surface there’s just input, no output. That’s ok for some applications (like turning your whole wall into simple controls for a radio), but what we really need is a thin transparent film that can also display images. Combined with the DMT a thin film display would turn any surface into a full-fledged touchscreen. Perhaps if the multitouch half of the concept works well, someone will develop the display half (we’ve already seen a few technologies that are headed in that direction).

Of course, we have no idea how well the concept will catch on until it actually hits the market. Certainly interactive window displays (as seen in the video) and wall controls are cool, but they may not find a customer base, especially if they are expensive. I wonder if LCD screen manufacturers will be willing to incorporate DMT into their existing models as a quick bootstrap way of turning every computer monitor into a touchscreen. That could be a really cool application that catches on very quickly.

I’m very curious about the embedded nanowires in the polymer film. Displax doesn’t manufacture this film itself (it has a partner for that) just the processing unit. Could the same technology that turns pressure into electrical disturbances generate electrical current? Could nanowires in polymers serve as ultrathin computer circuits? There are likely to be many applications for this kind of technology outside of touchscreens.

LCD TV + Displax = Touchscreen? Could be that simple.

For now, though, I’m just impressed with the wide range of places we might see these screens appear. It’s too soon to know if they’ll catch on, but they could turn every shop window into an interactive catalog for the goods inside. Your chair could know if you’re sitting on it, and could play relaxing music if your seat-signature seems too tense. Any table top could become a keyboard, or a piano, or a drum kit. The possibilities are endless. It’s likely to take years to develop the software for such applications, but the DMT hardware seems to be available now. If Displax, or some other cheap touchscreen technology, catches on, it will go a long ways towards “waking up” our physical world so that it looks for and responds to our input. I wonder what we’ll tell it to do?

[image/video/source credit: Displax]

Those white boxes are position sensors that allow the WIND system to turn your gestures into robotic commands.

Of all the ways you could command a robot, turning your body into a game controller sounds like the most fun. Future Robotics Technology Center (fuRo), part of the Chiba Institute of Technology, has developed an upper body suit that does just that. The Wireless Intelligent Networked Device (WIND) uses several small sensors, each with 3D positioning, to translate user motion into robotic commands. WIND communicates with a robot via Bluetooth signals, eliminating the need for a direct wired connection. All sensor information is controlled by a System in Package (SiP) core which consolidates a PC’s worth of robot command capability into a single chip. The fuRo system uses gestures, not one-to-one motion capture, to dictate commands. In other words the user doesn’t raise a hand when she wants the robot to raise a hand, she raises a hand when she wants the robot to dance. It’s a very cool looking control scheme when you see it in action. Check out the videos below to see for yourself.

Gesture controls are really hitting it big this year. We’ve seen gestures replace TV remotes – why press a button when waving your hand works equally well? The Acceleglove is a wearable device that translates movement to command a robot or record ASL. Yet the WIND is the first such system I’ve seen that uses the entire upper body to provide gesture controls. It increases the possible vocabulary of movements by a large amount. WIND also seems to be quite fun judging by the users in the video.

Plasticpals provided the following video which shows a live demonstration of the WIND and fuRo’s Morph 3 robot. Skip ahead to 1:05 to see how elaborate physical movements on the users part produce equally elaborate movement of the robot. However, the controller and robot movements are often quite different. Again this isn’t one-to-one motion control.

Which is, frankly, a little disappointing to me. If I shrug my shoulders I want my robot to shrug its shoulders not do a funny pose. Gesture controls are cool when it comes to computers and TVs, yet when there’s a humanoid shape in front of me I want it to mimic my human shape. That’s just a visceral reaction on my part, but I think it’s something most other robot enthusiasts would share.

Now, I don’t want that criticism to detract from what fuRo has accomplished, because they’ve done something great here. SiP technology could really change the scale of robotics, letting more control happen in less space. And WIND is a remarkable way to translate body movement into commands, and it could lead to some really cool innovations in human-computer interfaces. Getting the entire upper body into the command environment could free up one’s hands for separate tasks, or even allow body posture to signal for an automated response. Slouching? You could have your robot yell at you to straighten up. Have you begun to contort and grab your left arm? Your robot could call for aid because you’re having a heart attack.

It will be interesting to see where fuRo takes the WIND concept. Cyberdyne’s nerve signal sensing for its exoskeleton and Microsoft’s Project Natal highlight how full body motion capture could change robotics and gaming. WIND will have to find a niche outside of those arenas. Even if full body gestures don’t catch on, I suspect that the technology that enables WIND will find good use in other areas. Multiple position sensor signals routed through a central processor – sounds like the basis for the craziest Wii accessory ever.

[image credit: fuRo]
[video credits: fuRo, PlasticPals]

CES heralded gesture controls hitting the mainstream. Here, Toshiba's Cell TV gets put through its paces. Just wave your hand to select a channel from the floating sphere of television.

The modern age of electronics is turning everyone into royalty. We have court jesters (TV), scholars (computers), and scribes (office software) at our command. Soon, these servants will even be watching our slightest gestures and voice commands to heed our will. Hitachi announced its TV with gesture controls more than a year ago. Both it and Toshiba were touting “TVs without remotes” at CES 2010. Microsoft announced the XBox 360′s Project Natal, a system that uses the gamer body as a controller, at E3 last year. All of these technologies are scheduled to arrive at market by the end of 2010. Though the specifics differ among devices, each of these technologies would allow a user to simply gesture (sometimes speak) commands to their electronics. Change volume, browse through movies, or drive an arcade race car…all you need is your hands. The end of 2010 and the beginning of 2011 will see the first real attempt to get consumers to throw down their remotes and throw up their hands in jubilation. Will we want it? Check out some of the videos below to see what you’re buying.

Hitachi, Toshiba, and Microsoft are just the front runners on this trend. You could include Nintendo in that list as their Wii gaming system uses hand motions (through remotes) as a key ingredient in many games. And gesture technology itself is part of an even larger trend – the changing nature of the computer-human interface. Remotes, keyboards, mice, or game pads, most electronics translate button pushes into commands. The new paradigm for control will be one where our natural movements are harnessed to form a more intuitive way of expressing our desires to our devices. Wave your hand dismissively to change a channel, shake your phone to end a call, or play with blocks that represent files you want to move around. Eventually we’ll remove the physical interfaces altogether. Give projects like Braingate enough time and we could be commanding computers just with out thoughts.

Hitachi worked with Canesta and GestureTek to create its gesture interface. Toshiba has a remarkably similar system for its Cell TV. Both were on display at CES. The demonstration below, however, shows how these gestures are a first generation idea. They still require users to learn a new “language” of motions, albeit a simple one. (Note: the TV screen is blurry because its a new 3D model)

Microsoft’s Project Natal for the XBox 360 is a full body capture device. MS has announced it will be available by the end of 2010 (probably right in time for the holiday season) and will retail for less than $100. It also integrates voice commands. For video game applications, Natal seems better at translating user motion into actions on screen. Still, I would characterize it as imprecise.

In some ways, Project Natal is already pushing into the mainstream. Jimmy Fallon had a demo of the game system on his late night talk show way back in the summer of 2009. Notice the difficulty that he has with some of the controls.

It’s important to note that while big name companies have their brands at stake for these new devices, others have done much of the heavy lifting to develop the gesture recognition technology. PrimeSense and Canesta are responsible for most of the stereo cameras used for TV gesture recognition. GestureTek has created the software to translate perceived movements into commands. If gesture remote controls sell well for Hitachi and Toshiba, we’re likely to see them included in many other products thanks to these third party developers.

For those of us who grew up with remotes or video game controllers in our hands, there is something wonderful about interacting without them. I own a Wii and really enjoy waving the controller around rather than pushing buttons on it. I’ve had friends enjoy playing virtual tennis so much that they developed tennis elbow. At CES this year I even got to wave my hand at a TV with a gesture interface. It was fun. But in none of these situations do I feel I have the precise control I enjoy with my keyboard. Or with my TV remote. For someone who gets frustrated when it takes more than one push to “mute” my TV, I am worried by the loose and sometimes delayed reaction of gesture controls.

Gesture recognition may be a worthy step towards better human computer interaction, but if enough consumers share my worries it will change to just a fad. The overall trend towards intuitive controls, however, is something you can count on. The growing popularity of touch screens and accelerometers in smart phones is just one sign that consumers want a more natural way to command their personal electronics. We want our digital servants to not only know what we are thinking, we want them to be obedient to the exact details of our order. One day, devices will directly translate our thoughts into commands. Until then, we’re likely to see many efforts to create an intuitive and friendly set of controls for the technology around us. The end of 2010 will see gesture recognition’s attempt. Good luck to Microsoft, Hitachi, Toshiba, PrimeSense, Canesta, GestureTek and all the others. You’re going to need it.

[screen capture and video credits: Televisions.com, Microsoft, NBC, CBS Interactive, Prime Sense]

Lumino blocks interact with Microsoft Surface to extend digital information into a 3D environment.

It never fails: give scientists a new bit of technology and they’ll find a way to use it to play with blocks. Maybe it’s all the Legos we enjoyed as children. In any case, researchers at the University of Potsdam’s Hasso Plattner Institute have developed Lumino, a system of blocks that interact with Microsoft Surface. The table sized touchscreen has had many interesting features, but never true 3D manipulation. Lumino changes all that by letting Microsoft Surface see through the shapes it uses. Each block is identified by markings and these markers are transferred through lower level blocks to the touchscreen via embedded fiber optics. This lets a user stack up to 10 layers of Lumino blocks to create a 3D shape that can interact with Microsoft Surface’s software. Developers suggest it may have powerful applications in architecture and design. Check out the video from New Scientist after the break.

We’ve seen many different next generation human-computer interface (HCI) devices. Most focus on including a tactile/kinetic interaction to give humans a more intuitive means of manipulating digital information. Lumino does that and provides a truly dynamic 3D environment, something that with which other systems have struggled. The blocks used in Lumino do not contain any electronic or mechanically moving pieces, they are essentially just hard shapes filled with fiber optics. This simplicity sets Lumino apart from photoelastic techniques which rely on an added overhead camera. Still, watching the demonstration doesn’t leave me with the impression that I really need a Lumino system, or even Microsoft Surface. It will likely take several rounds of invention, redesigning, and marketing before these next gen HCI systems are sophisticated enough to provide an experience that is obviously superior in day to day applications to what we use now. When that happens, sign me up for some high tech building blocks. I’ll want to play, too.

[photo credit: University of Potsdam]
[video credit: New Scientist]

You don't need the remote, just wave your hand!

In some households, fighting over the TV remote is a raging nightly battle. In mine it’s more of a cold war detente. Either way, by the end of next year Hitachi (NYSE: HIT) may take the conflict to a whole new level. Working with Canesta and GestureTek, the Japanese electronics giant has created a line of television sets that will be able to recognize a viewer’s hand gestures. Instead of a remote control, you can just wave your hand in the right way to change channels or volume. Check out the video after the break to see Hitachi’s demonstration at CES from earlier this year.

From tablet PCs to iPhones, designers are giving us new ways to interact with our electronic devices. The future of the human-computer interface is likely to be much more tactile and intuitive than our current dependence on keyboard, mouse or remote control. With gesture controlled television, Hitachi and its partners aren’t just removing the necessity of a remote, they’re blurring the lines between the real world and the digital one. Right now, the space between you and your TV is just empty air, but in a few years it could be where you visualize a virtual remote control, or where you interact with your stereo, or the space in which you can clap to tell your laptop to go to sleep. Already, we’ve seen how any hard surface can be turned into a simple input device through acoustics. Imagine what will happen when every open space could be used as a digital interface. It could be an amazing way to integrate electronics into our lives, or it could be a horribly confusing way to crowd our personal space. Probably both.

GestureTek was responsible for creating the software that recognizes gestures and defines which gestures will control which functions of the TV. A little wave indicates you wish to make a command, a large wave scrolls through channels or options, pushing forward selects, a lowering of the hand scrolls down, and two hands can be used to pause playback. Twirling a finger or hand in a circle can vary an option (like raising the volume) and is reminiscent of the iPod dial. GestureTek has tried to use universal gestures, but there may be some variation across regions.

The Canesta hardware that makes the gesture TV possible uses cameras to capture a 3D image of the audience. This technology allows the TV to differentiate between your hand and a picture of a hand. The following Canesta video shows how the 3D image capture may be adapted to an entire entertainment system.

A TV that watches what you are doing is straight out of Orwell’s 1984. Understandably, Hitachi and its partners have tried to ease concerns that the television could become a tool of Big Brother. The Canesta cameras are sensitive enough to distinguish between individuals in a family and would allow each member to “save” their own settings on the TV. However, Canesta and Hitachi maintain that the cameras would not be sensitive enough to pick one user out among millions. The TV could be able to tell that you are the thin woman that likes action movies and makes broad gestures, and yet still not be able to identify you as Kate Beckinsale. Of course, I’m sure hard-line conspiracy analysts will find that guarantee to be cold comfort.

As cool as the Hitachi TV may be, I’m not sure it will be practical. You never know how consumers will respond to new interfaces until they live with them everyday in their homes. Still, even if waving your hands around doesn’t become the next big innovation in entertainment, we are certain to see other devices take advantage of rapidly improving gesture and facial recognition software. Our movements, and the space around us, are set to become the next creative space as the digital revolution continues. Which is great, but it’s unlikely to keep my remote control conflict from escalating. Compromising and watching the nightly news = mutually assured destruction.

[photo credit: Hitachi]
[video credit: New Media Geek RF, Canesta, Hitachi]

Rotating and tilting the Q2 Cube radio is all you need to do to change stations or volume.

As devices become more and more complex, they have the potential to become too difficult for humans to control. Hence we have the new paradigm in technology – simple interfaces for complicated machines. Exhibit A: the Q2 Cube. This new digital radio from Cambridge Consultants and Armour Group can stream internet radio stations but is controlled simply by tilting or rotating. If you rotate the cube onto four of its faces, the radio plays one of four preprogrammed radio channels. Tilting the cube back and forth alters volume. It’s a simple set of controls for a cool product that’s set to hit UK stores for the winter holidays. You can even sign up to win one for free.

Engineering an intuitive way to control a machine isn’t easy. Designers barely seem to agree on what a power-on button should look like (thank goodness for that circle with a line in it). The more powerful a machine becomes, the simpler the interface humans may need so that they can focus on managing the tasks performed by the machine. The iPhone has an absurd list of capabilities (acting as a drone remote control being my favorite) but it works using a simple touchscreen. Future human-computer interfaces may rely on humanity’s sense of kinetics to keep things simple. Why mess with a bunch of buttons and dials when you can just shake or tilt the machine to control it?

The shaking and tilting of the Q2 is measured by an internal accelerometer which lets it know when to change stations or volume. Unfortunately, the simple set of front-end controls corresponds to a rather complicated back-end. The four stations are preprogrammed into the cube via a PC by temporarily attaching a cable to the radio. An online account (hopefully free) is required to select the internet broadcasts, and the Q2 only works if you have a wireless router for it to connect through. Still, once you set the thing up, the radio is button and dial free.

A lack of knobs and keys may not seem a huge selling point for a radio, but it highlights the trend in electronic interfaces: less is better. All of our consumer technology seems to be getting smaller even as it becomes more capable. In many cases there’s simply no room for controls. We will likely see an increase in devices that read the user’s actions to determine how to act, as with scratch input or QB1 computer. For now, the Q2 Cube shows that the human-computer interface can work on a purely physical level. Don’t like the radio station? Just smack the cube and make it roll to a new one. I’ve been using that technique since I was a child.

BumpTop new multi-touch desktop lets you manipulate files using simple gestures.

Do you ever wish you could just reach out and move the icons on your computer desktop around the same way you do with pieces of paper on your physical desk? BumpTop has created a software system to allow you to do just that. Their version of a computer desktop has icons that appear as objects with weight and shapes that can be manipulated, letting their interaction seem much more intuitive to users. Today, BumpTop announced that it has taken the concept to the next level, and will provide multi-touch support for Windows 7 and MS touchscreen. CEO and Co-founder Anand Agarawala explains the original concept of BumpTop in a TED video after the break. Then, we have the multi-touch approach demo video right below.

So there’s really two products here, the current BumpTop system which you can upload onto your PC right now via their website, and a future touchscreen enhanced version that will likely be released to coincide with the adoption of Windows 7.  Both versions are trying to make files on your desktop more like real world objects, like so many other experimental human-computer interfaces we’ve discussed. The files can collide with one another, be stacked in piles, etc, just like pieces of paper. At the same time, you don’t want to lose the benefits of digital files: easy sorting, changing relative size of object on desktop, etc. The trick is to make the digital space act like a physical space in some areas, but still have the digital utilities where you need them. In the TED video you’ll notice that while Agarawala can use a mouse to throw files around like paper, he also can click to bring up menus that allow for sorting and other digital techniques. That sort of hybridization approach to human-computer interfaces is a powerful paradigm we are likely to see dominate the future of computing.

If Agarawala’s TED talk has you fired up to try the original (non-touchscreen) BumpTop, you’ll be pleased to know that a free trial version is available on their site. The fully fledged professional version costs around $30, and can be downloaded from the same page. BumpTop is also hoping to expand its user base through bundled packaging. Since August, the desktop software has been included when you purchase certain 3D graphics cards from HIS, Powercolor, and Sapphire.

BumpTop’s upcoming multi-touch approach, uses a set of defined gestures to control the way files are moved and changed. It’s sort of a precursor to a manipulation-language. Just as we’ve learned to use ‘ctrl’, ‘shift’, and other keys on a keyboard we could learn new ways to position our hands and move our fingers to augment the way we control our new real-world inspired desktop.

While BumpTop may seek to patent or otherwise control that gesture language, I hope that some version of it becomes too universal to be restricted to a single platform. As we move to computers without keyboards or mice, we’ll need a shared understanding of how to access digital processes through physical movements. BumpTop’s ‘scrunch’ move to put files in a pile works well, as does the ‘lasso’ gesture, which accomplishes the same thing. However, the same scrunch movement could just as easily have been used to crumple, delete, or shrink a file. The lasso gesture is similar to a common editors mark for removing a line of text. Without a shared gesture language that everyone agrees upon and dynamically controls, a user will have to rethink their movements for every new platform. Hopefully BumpTop will cooperate rather than compete with other multi-touch developers to form that language.

Making the digital world resemble the physical world is a powerful technique that will only grow more useful as digital information is infused into objects all around us. We’re entering a time when walls are becoming computer screens. As augmented reality takes off, intuitive interfaces are going to become a vital necessity to help users navigate their enhanced experience. Whether or not BumpTop becomes the model for such navigation, I’m confident that the human-computer interface of the future will be a fusion of the best parts of the physical and digital experience.

First up we have a video of  MIT’s David Merrill demonstrating a technology called Siftables at the 2009 TED conference.  Siftables are cookie-sized, computerized blocks you can stack and shuffle in your hands.  By arranging them in different configurations or tilting them at different angles you can do math, play music, spell worlds, pour virtual paint, and more.  The implications for hands on learning and manipulation of data are fantastic!  We have not seen any word on how/when this technology will be commercialized, but we hope it will be soon!

Next we have a technology for making music called Reactables.  By arranging and manipulating computerized blocks on a special table, musicians are presented with a completely new way of creating and interacting with music.  As seen in the previous video, Siftables are also capable of music composition, but reactables are unique in their singular focus on doing only music.  Whereas the siftables can perform many functions, the reactables are specialized for one task only, and in the coming years we can expect them to far outstrip the ability of Siftables when it comes to music.  Orginally created by Music Technology Group at the Universitat Pompeu Fabra in Barcelona Spain, Reactables have recently been spun off into a private company that is hard at work commercializing this exciting product.  For those that are really interested in this technology, there is a competing effort from Sony that may be of interest:

Kommerz from Austria brings us the mixed reality interface.  Using representative objects in the real world a person is able to manipulate objects in 3D space on a computer screen.  The possibilities for a new gaming interface look especially promising with this technology.  Check it out:

Thanks to Andreas for the above video.

This next demo from Sony has been around for many years, yet it is still very cool.  Why isn’t this technology finding a commercial market after all these years?  We have no idea.

Jeff Han from NYU demonstrates the capabilities of a multitouch interface at the TED conference in 2006.  Since then he has started a company around the technology called perceptive pixel.  This technology was recently used on CNN for presidential election coverage.

Speaking of multitouch interfaces, Microsoft has a technology called Microsoft Surface that is similar to Jeff Han’s technology, but in typical Microsoft fashion the company just doesn’t seem to get it.  Check out first a video from Microsoft that showcases the technology, followed by a hilarious parody from sarcastic gamer that shows how misguided Microsoft’s vision is:

The Khronos Projector is an interactive-art installation allowing people to explore pre-recorded movie content in an entirely new way. From the official site: “by touching the projection screen, the user is able to send parts of the image forward or backwards in time. By actually touching a deformable projection screen, shaking it or curling it, separate “islands of time” as well as “temporal waves” are created within the visible frame. This is done by interactively reshaping a two-dimensional spatio-temporal surface that “cuts” the spatio-temporal volume of data generated by a movie.”

Sixthsense from MIT is a technology that we have already covered in depth previously. Check out our detailed review for more information:

The world of interactive technology is literally exploding.  There must be several technologies we have overlooked in this review.  If you know of any that we missed, please let us know in the comments and we will try to add your suggestion to this post in an update.