Touchscreens and swarm robotics go together like geeks and video games.

Mark Micire’s PhD dissertation puts robotic control at his fingertips. The UMass Lowell student developed a command and control program for the Microsoft Surface touchscreen so that swarm robots can be easily guided. Watching Micire’s program in action makes it look like he’s playing StarCraft, only with real robots. Teams of bots can be color coded, groups can be selected by circling them with a finger, and robots can be commanded to move either individually or en masse. You can even manually drive a robot with a special pop-up interface. What’s the use for this multitouch control system? There are military applications, but Micire has a strong background in search and rescue robotics. Touchscreen swarms could be the next innovation in disaster relief. Watch Micire’s demo his system in the video below.

Swarm robots come in a huge variety of shapes, sizes, and capabilities, but they generally follow one strategy: many hands make light work. Bots are used together to divide and conquer a problem, overcoming challenges with a large number of workers. Many of the swarms we’ve seen are autonomous. In critical missions, however, robot AI may not yet be sufficient to find a solution to a problem. That’s why human guidance is still very important. Micire’s touchscreen program gives the operator various levels of control. This would allow humans to adjust their involvement as the case warrants. If robot autonomy can handle a situation, commands could simply instruct bots in a general way – go there, look for human bodies, report back. The system also allows for more direct commands – follow this route, etc. When direct human control is needed, operators can manually drive bots using the DREAM interface (Dynamically Resizing Ergonomic And Multitouch), as you can see at 2:23 in the video below.

Unfortunately, the demo above doesn’t show the system controlling physical robots. The bots on the screen are simulated. But Micire can control real world robots using the touchscreen, as you can see in the video below:

Micire’s work seem immediately applicable to fieldwork for robotics. It’s really no surprise considering his experience in search and rescue operations. It’s important to remember that robots already form a valuable part of S&R teams, allowing humans to explore areas that are too hazardous or difficult to reach. Case in point, here’s Micire’s video of work he did in Mississippi after Hurrican Katrina back in 2005

Of course, there are many applications for guided swarms besides disaster relief. Autonomous and guided drones are valuable assets in modern warfare, and advanced control systems like Micire’s could improve their usability. We’ve already seen how a similar technology (telestrators) are being developed by the US Air Force. Additionally, touchscreen controls could help swarms tackle industrial maintenance, exploration, or even surgery.

Swarm robotics holds a lot of potential, but most of the focus we’ve seen has been on the robots themselves. Micire’s work shows an interesting way in which humans can be readily inserted into the robot’s decision making process. In the near term that will allow these swarms to perform better as human intelligence still exceeds AI. Once autonomy out-paces human decision making we may still use such command systems as a means of providing over-arching control of our robots. It will be interesting to see if the potential of Micire’s program attracts any real-world applications in the years ahead.

…Maybe someone needs help fending off a zergling rush?

Video of Mark Micire’s complete PhD defense can be found here.
[screen capture and video credits: Mark Micire]
[source: UMass Lowell, Mark Micire]

Lumino blocks interact with Microsoft Surface to extend digital information into a 3D environment.

It never fails: give scientists a new bit of technology and they’ll find a way to use it to play with blocks. Maybe it’s all the Legos we enjoyed as children. In any case, researchers at the University of Potsdam’s Hasso Plattner Institute have developed Lumino, a system of blocks that interact with Microsoft Surface. The table sized touchscreen has had many interesting features, but never true 3D manipulation. Lumino changes all that by letting Microsoft Surface see through the shapes it uses. Each block is identified by markings and these markers are transferred through lower level blocks to the touchscreen via embedded fiber optics. This lets a user stack up to 10 layers of Lumino blocks to create a 3D shape that can interact with Microsoft Surface’s software. Developers suggest it may have powerful applications in architecture and design. Check out the video from New Scientist after the break.

We’ve seen many different next generation human-computer interface (HCI) devices. Most focus on including a tactile/kinetic interaction to give humans a more intuitive means of manipulating digital information. Lumino does that and provides a truly dynamic 3D environment, something that with which other systems have struggled. The blocks used in Lumino do not contain any electronic or mechanically moving pieces, they are essentially just hard shapes filled with fiber optics. This simplicity sets Lumino apart from photoelastic techniques which rely on an added overhead camera. Still, watching the demonstration doesn’t leave me with the impression that I really need a Lumino system, or even Microsoft Surface. It will likely take several rounds of invention, redesigning, and marketing before these next gen HCI systems are sophisticated enough to provide an experience that is obviously superior in day to day applications to what we use now. When that happens, sign me up for some high tech building blocks. I’ll want to play, too.

[photo credit: University of Potsdam]
[video credit: New Scientist]

Forearm sensors translate muscle movements into control commands for Microsoft Surface.

Researchers at the University of Washington, University of Toronto, and Microsoft Research have developed a system to control a computer through a device that reads muscle movement. Using eight sensors attached to the surface of the forearm you can now communicate basic commands by moving your fingers and hand in stylized gestures. The team of developers has adapted the new system to work with Microsoft Surface, the advanced table sized touchscreen. We’ve got a great demonstration video of the muscle control hardware interacting with MS Surface after the break.

It seems like every few days, a tech company finds a new way for us to control computers. These next generation human-computer interfaces all seem to have one goal in common: increasing the physical intuitiveness of computer control. In some cases, the physicality is expressly required by the device. The HAL cyborg from Cyberdyne relies on electromyography (EMG), just like the new muscle sensing control technology. However, tactile interfaces are becoming more popular purely as replacements for keyboards and mice, especially in casual environments like the Hard Rock Cafe in Las Vegas. For those of us who have adapted well to typing and point and click commands, the new physical interfaces may seem imprecise. To some extent they still are. Yet when paired with improved algorithms for speech, gesture, and facial recognition the new line of human-computer interfaces is getting ready to connect us directly to our digital world. Keyboards and mice, like so many middle men in our evolving economy, are being cut for efficiency.

Watching the demonstration video for the Microsoft Surface muscle-enhanced controls, I am struck both by the ease of use, and the lack of necessity. Do we really need a way to let each finger represent a different color? Is it that important to have pressure sensitive finger painting? Of course not, but clearly these are just first steps. On a standard computer, you have two areas: a control space and a display space. In touchscreen and other new mediums, the two areas have merged. Muscle controls are just one way in which we may be able to take advantage of that new overlap. Building on intuition, the simple pinches to pick up and flicks to undo could evolve into a gesture system which maximizes control of the device while being easy to use. It will be interesting to see how these gestures differ across the globe. Will everyone find flicking a common sense way to erase their actions?

Clearly the EMG controls have applications far outside of the Microsoft Surface platform. The research team presented some of those applications at the ACM Symposium on User Interface Software and Technology (ACMUIST). Their presentation claims that the technology’s recognition of gestures is fairly accurate in all sorts of conditions: 79% while making pinching motions, 85% while holding a mug, and 88% while carrying a weighted bag. Right now, the system requires eight sensors to be attached to skin of the forearm and hardwired to the computer. Future designs may require fewer sensors and take advantage of wireless signaling. We’ve got the ACMUIST presentation below.

I consider the EMG sensors, Microsoft Surface, and many other “next-generation” input devices to still be in their infancy. They’re expensive, not widely available, and in need of many iterations of refinement before they’d be competitive with current technology. Of course, they are also amazingly cool. Which is probably why some, maybe even most, of them will continue to be adapted and evolved in the upcoming years. I can’t wait to see what they grow into.

[screen capture and video credits: Microsoft, University of Washington, University of Toronto, ACMUIST]