Editor's Picks

Robot: I Now Have Common Sense. Engineer: Great, Go Fetch Me a Sandwich!

11,688 7 Loading

"Extra oil and vinegar please!"

Willow Garage’s gifted PR2 robot just got even smarter. Developers from the University of Tokyo and the Technical University of Munchen have collaborated to give it the ability to find an object it can’t see or isn’t even sure is there. In a way similar to humans, the PR2 can now go after an object by reasoning where it is most likely to be.

If you’re at someone else’s house and you need a bottle opener you’d probably begin by looking in the kitchen drawers as opposed to, say, the bathroom (candles, maybe, but not the bottle opener). By using “semantic search” the PR2 calculates the probability of finding the object in a number of places. If it doesn’t find it in the first place, it continues its search in other likely places.

In the following video, University of Tokyo researchers send the PR2 out for a sandwich. It looks in the refrigerator, doesn’t see one, then takes the elevator to the ground floor and orders one at Subway!

This sort of common sense is not so common among robots (and many humans too, but that’s beyond the scope of this article). Key to semantic search is knowing where sandwiches, etc are likely to be in the first place. For this reason, the researchers introduced probability maps that indicated the refrigerator and Subway are high probability sandwich locations. The maps can be updated say, if Subway closes, and like seemingly everything else modifications can be made from an iPad.

Semantic search is the latest creation from TUM’s Intelligent Autonomous Systems. The PR2 robot platform uses Robot Operating System-based software that TUM has already contributed to significantly. Semantic search can be combined with other code packets found in the treasure trove of ROS so the PR2 can cook you breakfast, fetch you a beer, or clean up after you in a smarter way. In a video we showed last year, the PR2 depended on a human to decide which cups and bottles on the table were to be cleared away for washing and which ones should stay. By increasing the specifications of semantic search the robot’s selectivity can be made finer so that, for instance, any cups that are half-full should remain.

If robots are going to work with us and for us they’re going to need the kind of common sense that semantic search gives them. If we have to tell the robot what room, what refrigerator, what shelf and what can of beer it needs to fetch we’re apt to do it ourselves.

And that’s just no fun.

video: sandwich

Peter Murray

Peter Murray was born in Boston in 1973. He earned a PhD in neuroscience at the University of Maryland, Baltimore studying gene expression in the neocortex. Following his dissertation work he spent three years as a post-doctoral fellow at the same university studying brain mechanisms of pain and motor control. He completed a collection of short stories in 2010 and has been writing for Singularity Hub since March 2011.

Discussion — 7 Responses

  • why06 October 8, 2011 on 10:28 am

    Omg that’s awesome. It would be nice if these maps of information could start to be built programmatically. Like just from things it hears it can start to reason that subway is closed or other things. And link information from other sources. Like find out if there are other subways and use google maps to find its location or something.

  • Victor F Magana October 8, 2011 on 11:13 am

    So millions of dollars thousands of robots, millions of man hours and a 3 year old is more functional… Now I see why politicians can’t do much with millions either.

    • goldterc Victor F Magana October 9, 2011 on 9:06 am

      But you can create a thousand 3 year olds just like it, the hardware is waiting for software to catch up. Google have their self driving car, put this in the back and it could deliver parcels from amazon , it could even pick the items in the warehouse and package them up, put them on a pallet and the google delivery truck would move them to a distribution center to be unlaoded by it’s brothers/sisters/cousins.

    • nick Victor F Magana October 9, 2011 on 3:22 pm

      a 3 year old took a lot more time and resources to evolve than the robot took to be created

  • Joe Nickence October 8, 2011 on 3:34 pm

    Considering that the bot is notoriously slow, I’m guessing the guy ate about an hour later. It’s nice for proof of concept, though.

  • Homer October 11, 2011 on 10:20 am

    How did the robot order the sandwich? Did it apeak? Or did the engineer call ahead?