This X-ray/CT scan shows the placement of electrodes over the temporal lobe that scientists used to decode what the patient was hearing.

The day we can scan a person’s brain and “hear” their inner dialogue just got closer. Scientists at the University of California, Berkeley recorded brain activity in patients while the patients listened to a series of words. They then used that brain activity to reconstruct the words with a computer. The research could one day be used to help people unable to speak due to brain damage.

It’s not every day neuroscientists get a chance to record activity from the human brain. A group of 15 patients suffering from either epileptic seizures or brain tumors were already scheduled to undergo neurological procedures. The patients, all English speaking, volunteered for the study. After neurosurgeons cut a hole in their skulls, the research team placed 256 electrodes over the part of the brain that processes auditory signals called the temporal lobe. The scientists then played words, one at a time, to the patients while recording brain activity in the temporal lobe.

The auditory features of a sound can be characterized by what’s called a frequency spectrogram that measures the strength of different frequencies within the sound. The scientists hoped to use the pattern of brain activity while the word “partner,” for example, was played to generate a second spectrogram. Were it perfect, this process, called “stimulus reconstruction,” would reconstruct a frequency spectrogram identical to the original. However, because the temporal lobe is only one of several brain regions that process sound, the scientists did not expect an exact reconstruction.

Brian Pasley, post-doctoral fellow and lead author of the study, devised two different computer models for the stimulus reconstruction. Each was created according to different assumptions about how the brain processes sound. One model outperformed the other, enabling the computer to reconstruct the original word 80 to 90 percent of the time.

A comparison of original and reconstructed frequency spectrograms for a test word.

The study was published recently in PLOS Biology.

That’s music to the ears of people who can’t speak because of brain damage. Strokes or neurodegenerative diseases such as Lou Gehrig’s disease can leave people’s language centers damaged and impair their speech. A critical link between the current study and potentially helping these people is the idea that hearing words and thinking words activate similar brain processes. There is evidence to suggest that this is indeed the case, but more research is needed to work out exactly how perceived speech and inner speech are related. Even so, the current study lends hope to a potential treatment. “If you can understand the relationship well enough between the brain recordings and sound, you could either synthesize the actual sound a person is thinking, or just write out the worlds with a type of interface device,” Pasley told the Berkeley News Center.

Just last year researchers used brain implants to improve memory in rats. Others used an implant to help paralyzed rats walk again. The current study paves the way for yet another domain in which neural interfaces could one day be used to improve our lives.

[image credits: Adeen Flinker via UC Berkeley and modified from PLOS Biology]
image 1: brain waves
image 2: figure

"Sensors that can detect the weight of a dead butterfly? Artificial skin must be just around the corner!" ...Um, no.

Even the coolest lab work is often years from practical applications. That’s what the media seemingly forgot in regards to recent developments in ‘artificial skin’. Researchers at UC Berkeley and Stanford have independently created thin layer sensors that can detect small changes in pressure. Both UC Berkeley and Stanford had their work published in Nature Materials and the two approaches to advanced pressure sensors have been covered by the BBC, Reuters, IEEE Spectrum, ABC, and the Wall Street Journal. The thin layer materials have been hyped as having sensitivity close to that of human skin and there’s been talk that they could provide the means for artificial skin for prosthetics or robots. While the work in these two projects is interesting, it’s many years away from any marketable projects and there are major hurdles they would have to clear before they could be used in the field in any application. Pragmatism, however, has been pushed aside in the glee over watching sensors detect a butterfly corpse and creating neat pictures of artificial skin gloves. Check out the video below to see what I mean. These are cool pressure sensors, but they aren’t skin-like yet.

Stanford’s sensor is composed of electrodes on either side of a thin film of rubber. These pressure sensitive layers have been constructed in the shape of tiny pyramids, allowing the sensor to spring back after being compressed. This has given Stanford’s material both extreme sensitivity and quick recovery. They’ve tested the system by detecting the weight of dead insects such as flies and butterflies. This has lead many media outlets to fill their stories with ideas that the sensor could let someone feel an insect land on their artificial limb. You can almost see how researchers in the video below are being lead to make these kinds of comments:

The work out of UC Berkeley is no less impressive for its accomplishments in energy use and flexibility. Their ‘e-skin’ sensor utilizes germanium and silicon nanowires to form a thin film transistor. This TFT runs on less than five volts, and can handle being bent 2000 times on small radii folds (<2.5mm) and still function. As its name implies, the e-skin has been considered as a means of providing human-like touch sensitivity for prosthetic limbs as you can see in the computer generated image from the lab.

UC Berkeley's e-skin shown as a covering for an artificial limb sensitive enough to hold an egg. We're years from such applications.

Both of these projects are remarkable feats in sensor technologies, but I object to categorizing these sensors as artificial skin, especially for prosthetics. Such an association glosses over the limitations we have in connecting mechanical sensors to our nervous system. Even though these sensors can mimic the sensitivity of human skin under the normal range of human pressures (0-15 kPa or so) how are we going to transmit that information to your brain? Our most advanced sensing prosthetics handle just a few points of touch, not the hundreds or thousands of points that you’d associate with a full hand like one imagined in the image.

True, in the future we’re likely to be able to improve the way we connect mechanical/electrical signals to our brains. Or maybe we just want to use these types of artificial skins for robots, which have an all electrical ‘nervous system’ anyway. Well, even then, I’m worried that media coverage ignores how few of these sensors have actually been made. The Stanford sensor looks like a single hand made pressure cell. Even the UC Berkeley TFT is made in a fairly small array (7×7 cm with 18×19 pixels). Mass producing these sensors is not going to be trivial. We’ve seen projects for robot skin before, some of which, like the QTC sensors, have been under development for years. They’re still working on getting these systems ready for field work and real world applications.

You know, if headlines for these projects read “California researchers develop advanced pressure sensors” I probably wouldn’t be objecting at all. Heck, I’d probably have written my own effusively praising article that alluded to the idea that such sensors could one day be used in artificial skin for prosthetics and robots. But calling these projects ‘artificial skin’ right off the bat seems false to me, and a bad case of hype. Yes, these sensors may have human like sensitivity in the pressures they can detect, but that’s a far cry from actually being used to create artificial skin. I applaud both UC Berkeley and Stanford for their remarkable work in pressure sensors, but please don’t expect either of these projects to evolve into a marketable skin-like product for many years to come.

[image credit:Stanford University, UC Berkeley]
[video credit: Stanford News/Jack Hubbard]
[source: Stanford News, Berkeley News; Takei et al and Mannsfeld et al, 2010 Nature Materials]

State authorities told Berkeley it couldn't give students personalized results for even the most trivial of genetic tests.

UC Berkeley sparked a bit of controversy earlier when they announced they would be giving 5000 incoming freshman the option of taking a simple genetic test. Now, the “Bring Your Genes to Cal” program has been curtailed by state regulators. As we discussed earlier, students were asked to provide DNA samples that would only be analyzed for three genes – those related to how humans process alcohol, lactose, and folic acid. About 600 or so have already participated. It was meant to be an interesting shared experience to spark discussion about personalized medicine. The California Department of Public Health decided, however, that the project wasn’t education but rather the rendering of medical services to students. As a result, UC Berkeley has decided to deny students access to the individual results of their tests. Instead, aggregated data will be presented. The personalized medicine discussion is going to be a lot less personal than planned. I understand that genetic testing is very far from an exact science at this point, but when are we going to stop being afraid of our own DNA?

Berkeley’s announcement comes on the heels of a long difficult year for personal genomics. The FDA has announced it will begin tight regulation of the industry, and already shut down one attempt at bringing personalized DNA test kits into pharmacies.  But the modification to the university’s ground breaking program isn’t framed in terms of the genetic industry, it’s supposedly about labs.

The Clinical Laboratory Improvement Amendments (CLIA) are federal laws regulating medical diagnostic laboratories. Because the original intent of Bring Your Genes to Cal was to give students access to their own results, the California Department of Public Health decided that the university was in fact giving them viable medical information and thus had to fall under the strict guidelines of CLIA. Berkeley is using it’s own genotyping lab which is generally exempt from CLIA. As such, Berkeley was given the options of either sending the DNA out to commercial labs (way too expensive to be a viable option) or to deny students access to personalized results. According to online statements from Berkeley, the university is completely complying with the Department of Public Health’s decision, but laments the hamstringing of the program.

In fact, this may have been a decision the university saw coming. They had sent a letter earlier in the summer to the Department of Public Health outlining how they believed the Bring Your Genes to Cal program was exempt from statutes such as CLIA because it is educational in nature. Berkeley also argued that students were only receiving genetic results on genes that had little to no medical value. Clearly the Department of Public Health didn’t agree.

As cool as the Bring Your Genes to Cal idea could have been, there are bigger concerns here. The California Department of Health’s interpretation of federal and state law has the potential to limit genetic research in public universities throughout the state. Unless academic testing is given broad exemptions from commercial regulation it will be strangled by costs and bureaucracy. We can ill afford to limit public research into genetics, especially in a state and university system that has been one of the emerging leaders in genetic research.

Honestly this entire situation pisses me off. I understand federal and state regulators wanting to protect consumers from a branch of medicine that is still full of more questions than answers. As we’ve discussed before, personalized genomics has had its problems. At its best current DNA testing is mostly about broad generalizations at this point. The concrete recommendations it can make for healthcare are few and far in between. Clearly this case was about more than that. It was wrapped up in concerns about lab standards and medical services. Still, it sets a very poor tone for the future. After all, we’re talking about genes for lactose intolerance, not Alzheimer’s! What happens when we want to learn about the really important genetic factors in our lives? Even if Berkeley was going to give students insight into serious medical conditions, early studies indicate people take negative results from such tests very well. Humans are not so scared of knowledge as people seem to think. Our governments don’t need to protect us from our own DNA.

[image credit: modified from KQED.org]
[source: UC Berkeley "On the Same Page"]

Willow Garage just announced the winners of its PR2 Beta Program. Time for a party.

Today is a good day to be a robotics engineer. Not only is it the first day of ICRA, it’s the day that Willow Garage (WG) officially announced which research groups will be part of their PR2 Beta Program and receive one of their very own PR2 robots to work with. I had a chance to talk with Willow Garage and get some insight into the project. While they originally conceived of parting with just ten of their unique open source PR2 platforms, Willow Garage had such an amazing pool of formal proposals (78 in total) that they had to crank it up to eleven. That’s about $4.4 million worth of robotic equipment that WG is providing for its winners without cost. For the next 24 months, or so, these institutions will use the PR2 to develop ground breaking software and, because all of it will be shared via open source licenses, that code will be available to robots everywhere via ROS. Millions of dollars invested, but untold riches in new open source robotic software ready to be discovered.  Read on to learn about some of the amazing projects that the P2 Beta Program will enable.

Jorge Cham of PhD comics did a great write up of Willow Garage's vision. He also begged for a free robot. I would have done the same.

I am an unashamed fan of Willow Garage’s work, not just for their humorous Warhol-esque posters, but for their dedication to making the future of robotics an open source one. Robotics is a detail driven, and technically challenging industry. Individual labs are often required to build their hardware and software from the ground up before they can even approach the visionary ideas that inspired them to start their work. Too much time is lost “re-inventing the wheel“. Open source can change all that. By sharing innovations institutions give each other the tested ingredients they need to build their programs quickly. Willow Garage’s PR2 Beta Program takes it even further. Not only are they fomenting the development of open source code, they are actively giving researchers world-class hardware to test and refine that code. This isn’t just keeping everyone from having to re-invent the wheel, it’s handing them a car. Who knows where they’ll go with it.

Here’s the official list of PR2 Beta recipients:

• Albert-Ludwigs-Universität Freiburg with the proposal TidyUpRobot
• Bosch with the proposal Developing the Personal Robotics Market: Enabling New Applications Through Novel Sensors and Shared Autonomy
• Georgia Institute of Technology with the proposal Assistive Mobile Manipulation for Older Adults at Home
• Katholieke Universiteit Leuven with the proposal Unified Framework for Task Specification, Control and Coordination for Mobile Manipulation
• MIT CSAIL with the proposal Mobile Manipulation in Human-Centered Environments
• Stanford University with the proposal STAIR on PR2
• Technische Universität München with the proposal CRAM: Cognitive Robot Abstract Machine
• University of California, Berkeley with the proposal PR2 Beta Program: A Platform for Personal Robotics
• University of Pennsylvania, GRASP Laboratory with the proposal PR2GRASP: From Perception and Reasoning to Grasping
• University of Southern California with the proposal Persistent and Persuasive Personal Robots (P^3R): Towards Networked, Mobile, Assistive Robotics
• University of Tokyo, Jouhou System Kougaku (JSK) Robotics Laboratory with the proposal Autonomous Motion Planning for Daily Tasks in Human Environments using Collaborating Robots

You’ll notice that four of these eleven groups are international. That’s more than Willow Garage originally anticipated, but it demonstrates the scope of the talent that sent in proposals for the PR2 Beta Program. If anything, that range of applicants already made WG’s difficult task of selecting recipients that much more challenging. Narrowing the list down from 78 proposals wasn’t easy.

The PR2 is one of the coolest robots on the planet. How do you decide who gets one?

The criteria for selection reflects Willow Garage’s own philosophy. A strong preference was given to those institutions which have shown a commitment to open source robotics. After all, WG is one of the driving forces behind the robot operating system (ROS), which will serve as the means of disseminating all the new code that the PR2 Beta Program will generate. WG also had a preference for groups and proposals that would potentially give a large number of students access to the PR2 robot. Looking at the press release from Willow Garage, you’ll see that many of these proposals (MIT, USC, etc) involve several labs/groups at the same institution. One of the points of the PR2 robot is for it to allow researchers to quickly test and perfect their ideas. That makes it ideal for large groups with many different developers looking for quality time with the robot.

Willow Garage has a strong commitment to develop robots that can assist and work with humans. Many of the projects they selected involve getting the PR2 to not only perform valuable tasks, but to do so in a human environment. Even those projects which do not expressly propose to interact in the people-sphere are aimed towards creating robust and practical applications that will serve humans well. There’s no doubt that Willow Garage is still the company looking to build a personal robot.

While there are some big name institutions in this list (MIT, JSK, Stanford, etc), there are also some labs still making names for themselves. Peter Abbeel, fresh from getting the PR2 to fold his towels, is a young professor and part of the group that wrote the proposal from UC Berkeley. In fact, looking at the list as a whole, you can tell Willow Garage was also selecting for a diversity of strengths. There’s a good selection of experience, enthusiasm, and areas of expertise.

I would have liked to see at least one complete unknown on this list. A random hacker space or amateur robotics engineer (maybe a Lego Mindstorms guru?) would have added some interesting flavor to the mix. Willow Garage did say that they received some great letters of intent (the first round of applications) from groups that were off the radar, but that few, if any, had the resources available to create the formal proposal. That’s just an artifact of our times, I think. Hopefully the years ahead will reveal more low-budget yet dedicated developers finding the means to take advantage of programs like this one.

UPenn's GRASP lab will look to change the PR2's hands...while it's working.

It’s hard to pick a favorite among these eleven, and hopefully they’ll all produce some amazing work in the next two years. I was told that Bosch has been showing some really amazing commitment to the project for such a big corporate name. Not only did they convert to ROS about a year ago (gasp! an open source international company?), they’re also sharing hardware with the rest of their PR2 Beta group. Advanced sensors, including “robotic skin” will be distributed to some of the other labs. (That skin may be similar to the product and program we saw coming from Peratech). UPenn is also working on some hardware adjustments, looking to develop a system by which the PR2 could change its claw to various auxiliary appendages carried in a toolkit on the robot. I’m personally excited to see how UC Berkeley will build on previous work with towel folding – their proposal calls for the PR2 robot to do a full load of laundry, assemble furniture from IKEA, and to learn a task by watching (as we’ve seen with other robot platforms).

It’s important to keep in mind that none of these teams will be working in a vacuum. The code produced will be shared with the greater open source community, just as these researchers form tighter collaborations amongst themselves. Willow Garage will soon be hosting a conference so that the students and group leaders can meet before the program launches. During the next two years, there will be regular check ins, exchanges of information, and fun interactions between labs, robots, and students. The professional relationships built here may pay dividends far beyond the PR2 Beta Program. In a way, this project isn’t simply building the next round of software for robots, it’s helping to create the next generation of robotics engineers with first hand experience of the advantages of open source.

Willow Garage seems hesitant to speculate on how the PR2 Beta Program will develop beyond this first round. ROS and other open source systems develop at their own pace and in their own unpredictable directions. Still, one hopes that these first two years will see such amazing successes that Willow Garage will have no choice but to renew and expand the program. It’s many years (or decades) down the line, but I would love to see an open source robot in every high school around the globe.

Of course, for those of us looking to the future, it’s not enough to simply anticipate the next round of technological innovations, we must also try to understand the mechanics that shape those innovations. Open source robotics , as enacted by Willow Garage and many others around the world, is a time saving and effort-maximizing paradigm. Researchers sharing their work openly will build more quickly, and with more robust results. This is a clear path towards an acceleration in robot technology. Not only could that path provide us with amazing robots far sooner than many anticipate, it will do so in a way that fosters cooperation on a large scale. That’s important if we want to build a better world, and not one simply filled with robots. So good luck to all the recipients of the PR2 Betas – your proposals all sound great, and I can’t wait to see them come to fruition. Though, as those at Willow Garage have said, what will be more amazing is what we don’t anticipate.

[image credits: Willow Garage, Jorge Cham via Willow Garage]
[source: Willow Garage (Press Release and interviews)]

UC Berkeley got Willow Garage's PR2 robot to fold towels perfectly. Rosie Jetson, eat your robot heart out.

Robots just got roped into doing some light housework. Researchers at UC Berkeley used Willow Garage’s PR2 robot to fold towels. The UCB programming used some innovative visual scanning allowing the PR2 to pick up a towel, find its corners, and fold it on a table perfectly. According to the paper presented at the 2010 ICRA, the robot successfully completed 50 out of 50 attempts to fold a single towel, and also folded 5 out of 5 towels when they were presented in a group. Is watching a robot do laundry really that exciting? Hell yes. We have a personal robot actually performing personal and useful tasks. It’s not dancing. It’s not welcoming you to an expo. It’s doing real work. That’s amazing. But you know what, forget all that, too! You know why this is really great? UC Berkeley used a Willow Garage robot to develop their own sophisticated robotics program. That validates the whole premise of the PR2 – faster development by letting researchers use a common platform. Score one for open source robotics!

I’ve been cheerleading for Willow Garage for a while now. Their approach to robotics exemplifies the new paradigms of innovation in the 21st century: distributed development, free exchange of information, and rapid leveraging of past successes into new prototypes. By building the PR2, Willow Garage essentially removed the burden of developing hardware from the UCB team. This let them skip the tedious creation of their own system and focus on the particular innovation they wanted to develop: a vision based detection system that can use geometry to overcome complex variations. Now, assuming that UCB will upload their code into the open source community, others can build on the work from there. This is how the field of robotics can be accelerated, and this is the deeper reason behind the celebration of this development.

Of course, watching robots fold towels is pretty frakkin’ awesome just on its own. The UC Berkeley team, under Professor Pieter Abbeel, has created a great algorithm for the PR2. It picks up a randomly folded towel it’s never seen before and twirls it until it finds a corner. Then it grasps that corner and finds the next until all corners are accounted for. Once the corners are identified the robot folds the towel and stacks them on a table. It’s elegant in its simplicity, complex in its visual recognition, and fun to watch when it’s sped up:

Just to show how robust the system is, here’s a challenging starting point with the towel partially folded and twisted in a complex way.

As cool as this project is, there’s obvious room for improvement. Speed is clearly an issue, but that may improve with processing power. Also, while the PR2 has a perfect score on successfully folding towels, this does not mean that it never committed an error in its trials. As mentioned in the ICRA paper, for 28 out of 50 towels the robot completed its task without a misstep, but in 22 trials there were some hiccups. These errors included missing a grab (16 trials), thinking it had grabbed a corner when it hadn’t (5), thinking it had not grabbed a corner when it had (3), incorrectly gripping a corner (4), unable to compensate for twists in the towel (3), and needing a complete start over (1). In each case though, the PR2 was able to continue on its algorithm (or start over) despite these errors and successfully complete its task. To me, that just means that Abbeel’s programming is more impressive because it’s robust.

No matter how rugged the PR2 and the towel folding code may be, I doubt that we’ll use humanoid bots for household chores. Specially crafted appliance-style robots can handle these tasks more efficiently and cheaply. But projects like this really demonstrate that personal robots can be workers as well as companion bots. As these machines become more self-sufficient (the PR2 can already plug itself in) they may be able to bring automation into places we’ve never seen it before.

Overall, I’m really excited about how the PR2 will advance personal robotics. It’s already helped UCB with a cool and novel application. Just think of what researchers may accomplish with this bot in the future. Don’t forget that Willow Garage is giving away 10 of these robots for free! This is a great time to watch how bots are going to evolve to take on a larger role in our lives. The next few years are going to be amazing and I can’t wait to see what UC Berkeley, Willow Garage, and the PR2 are going to do next.

Open source robotics for the win!

[image credit: Maitin-Shepard et al, ICRA 2010]
[video credits: UC Berkeley]
[source: UC Berkeley, Maitin-Shepard et al, ICRA 2010]

Inside this fMRI machine a test subject in Kyoto is having his mind read to determine which image he sees.

If you had to nominate one modern technology as a mind reading device, the fMRI looks like a good bet. By measuring blood flow fMRI can track activity in your brain, and this opens the window to your mind – it may even allow us to figure out what your eyes are seeing at any given moment. The ATR Computational Neuroscience Laboratories in Kyoto, Japan is able to show a geometric pattern to a test subject and then have a computer program recreate that image by analyzing brain activity gathered by fMRI (NIPS 2009). Scientists at UC Berkeley have used fMRI to study the visual cortex to encode images as brain activity and decode brain activity into images. In other words, for a given image they know how your brain will react, and for a given brain reaction they know the image that would cause it. Researchers at UCB have even managed to do the same with video – their decoding system can create a rough facsimile of what a subject was watching at the time. This is incredible! I had a chance to talk with Jack Gallant of UC Berkeley about these attempts to see what the brain sees. While this technology is still in its very early stages, the work already finished is truly astounding. Check out a video discussing ATR, and pics of research from UCB after the break.

fMRIs have been used in several different attempts to “read minds”. Carnegie Mellon is working on a system that can track how your brain responds when you see different words. When you read “apple”, the CM fMRI can measure your brain activity and know you weren’t looking at “badger” or “football”. Similarly, Microsoft Research has used EEG to categorize which kinds of images you are watching (an animal versus a face, for instance). They may use the technology to help automatically tag images at high speeds. Work at ATR and UCB, however, is a step in another direction. These are the groups which are asking: “if I can monitor your brain, can I reconstruct the image that was on your retina?” They are actively pursuing a means to take brain activity and translate it into something other humans could look at and understand.

The work at UCB, headed by Jack Gallant, has been underway for several years. He, Thomas Naselaris, and colleagues determined how a random image from a database of thousands could be viewed by a subject and then identified by a computer program that monitored brain activity through fMRI (as published in Nature, 2008). In September of 2009 they took that work one step further and developed the means to use semantic, structural, and prior information to accurately reconstruct an image (as published in Neuron). Were you staring at a house on a river? UCB’s computer program can use structural and semantic clues to figure this out and choose among a set of prior images which agree with the one in front of you. It may not be the same house on a river, but it will look close.

The images on the left were shown to test subjects at UC Berkeley while their brains were scanned by fMRI. Then a computer program used its understanding of how the brain codes structural and semantic information to guess which image, among thousands best fit the activity it saw in the fMRI (see on the right). While the left and right images aren't identical, they all agree semantically and structurally. A similar process has been shown to work with video.

The semantic input is an interesting twist. As with the work done at CMU and Microsoft, UCB’s work shed some light on the function of voxels (volumetric pixels, basically a 3D section) in different locations in the brain. Groups of voxel in the brain could be used to process a particular kind of visual structure, another group could be used to categorize a particular set of objects. While Gallant is adamant that fMRI is actually a clumsy tool (“we only work with it because it’s the best we have”) it can help scientists label patterns of voxels as coding for different kinds of images. Look at a face and one part of your brain lights up, look at a truck, and another part is activated. Get enough such semantic voxels labeled, and you’ve got a good tool to help identify any image you might show a subject. It seems likely that the first mind reading machines of the future will need to use such voxel-based semantic clues to help them guess what you’re seeing.

What I really find exciting about the research at UCB, is that they’ve already started to decode moving images. As reported at the Society for Neuroscience, Shinji Nishimoto (part of Gallant’s team) showed test subjects a long series of videos (hours of DVD trailers) while recording their brain activity with fMRI to understand how the brain encodes video data. A computer program then took this encoding process and used it to guess how the subject’s brain should respond to other videos (taken from YouTube). According to Gallant, the total length of this YouTube footage was equal to about 6 months of continuous video. That’s a lot of “Charlie Bit my Finger!”

So now you have a computer program that knows pretty well what each video in its collection should do to someone’s brain if they watched it. What do you do next? You show a test subject a completely new video (“the target”). The computer reads their brain and says, which of my video clips best fits this new activity? It finds the best matching one second clip of video and puts it into a compilation. As the subject keeps watching the target video, the computer is assembling this Frankenstein’s Monster of clips. The end result is a movie montage that jumps back and forth among thousands of videos. That montage, surprisingly, gives a fairly accurate portrayal of what the test subject saw in the target.

According to Gallant, fMRI is really limited in its spatial and temporal resolution. You can’t get much better than one second, and the voxels aren’t as small as neurons. Still, the fact that we can scan a brain and reconstruct what they are looking at in any form is astounding. Whatever technology replaces fMRI with better resolution is only going to increase the accuracy of these reconstructed images.

As always when computers are reaching towards your brain, the fear of invasion of privacy becomes paramount. We’ve already seen how governments are planning to use brain scans in security checks, could they start judging us for what we see? Could they even know what we are dreaming?

Possibly yes, but not yet. Brain scanning science is still in its (rather lengthy) infancy. And fMRI, the best technology at this point for reading brain activity, requires you to hold perfectly still in a huge machine with powerful magnets. Not the sort of thing you could secretly install in someone’s home without them knowing. One day, though, it’s possible we’ll engineer a device that can really capture exactly what you are seeing (or imagining) and record it. Memories could be recorded directly. Maybe even more abstract concepts could be observed – ATR is working to categorize visual art by the way that it stimulates brain activity. Once we get machines to look inside our brains its only a short hop to getting our brains linked together. Borg, here we come!

[image credit: UC Berkeley]
[screen capture and video credit: NTD World News]
[sources: Nature, Neuron, Jack Gallant, ATR Website (Google Translated), NIPS]

Like a cockroach, DASH is fast and robust.

Have you ever tried to catch a running cockroach? They’re fast little suckers, and resilient too, which is why scientists at the University of California Berkeley modeled their latest robot after them. DASH or Dynamic Autonomous Sprawled Hexapod is a six legged robot made from cardboard and polymer. It’s the size of your open hand, weighs just 16 grams, can run up to 1.5 m/s, and survives falls of 28 meters without damage! This cockroach bot is really something to behold. Check out the IEEE Spectrum video after the break and watch how DASH survives the fall from the top of a building and keeps running.

As robotic systems become increasingly complex, they are able to mimic real world creatures. There are toys like the hexbugs, and more advanced robots like StickyBot that exhibit animal skills and/or behavior. The reverse is also true, we’ve seen robotics used to help explain developments in evolution. This interplay between the sciences is important stuff. When the advances in one field are applied in another the rate of new discoveries increases. Feedback loops between biology and robots may be one of the means by which we develop the next generation of genetically engineered machines and life-like automatons.

Watching DASH in action, I’m stunned by how simple yet powerful the robot is. The oar-like motion of its legs is rudimentary, but it allows the bot to travel at 15 body lengths per second. Humans would have to run at 65 mph (~100km/h) to match this feat and we have much more complicated gaits. I’m equally impressed by the upwards scramble at 1:56. Can you imagine running at a 6 foot wall and just popping over it?

UC Berkeley’s Biomimetics Lab, where DASH was created, is also interested in the way that robots are built. To that end, DASH has one of the coolest constructions I’ve seen in a robot that performs this well. First, the materials used for the chassis are simple: cardboard and polymer. Just one DC motor for forward motion and a smaller servomotor for turns keeps the bot weight down (16.2 grams is next to nothing!). Second, the build time (less than one hour) is ridiculous for a hand made robot. Considering DASH’s simple construction, I wonder if they’d be willing to design a version that could be made on a 3D printer like MakerBot. Finally, the design is absurdly robust. I’ve seen some robots that you can toss around, and robots that can jump a wall, but none that can free fall for 25+ meters onto concrete and then walk away.

As cool as DASH is, I’m not sure that we’ll see much more of it. Upgrading to using a carbon fiber body would make it more resilient, but I don’t know that we need a resilient cockroach bot. Instead, I think that DASH is going to be an inspiration piece. It’s a robot whose design, construction, and performance will help engineers create the next generation of industrial and research machines. We could see smaller versions become part of a robotic swarm, or larger versions act as hazardous environment probes.

As engineers continue to adapt animals into robots, there are all sorts of amazing possibilities. DASH runs as fast as a roach, I wonder if there will be a bot that jumps as high as a flea, or that has the proportionate strength of an ant. Certainly we could use a beetle robot to deal with all the… dung… going on in our world.

[photo credit: UC Berkeley Biomimetics Lab]

[video credit: UC Berkeley and IEEE Spectrum]

The CellScope combines micrscopes with cell phones to help diagnose diseases in remote locations. Seen here is a low magnification prototype.

I think we need to stop calling them cell phones, because our hand held devices are starting to have more capabilities than Batman’s utility belt. Controlling robots, projecting images, depositing checks, augmented reality, not to mention internet, GPS, and cameras… and now we can add microscopes to the list. Prof. Dan Flectcher and his team out of UC Berkeley are developing the CellScope, a microscope assembly that will easily attach to a standard cell phone and allow you to take up close images of skin and blood samples. While that sounds like just another cool iPhone trick, it has huge implications for fighting diseases in the Third World. Check out Fletcher’s explanation video from Popular Science after the break.

Tuberculosis (TB) and Malaria kill millions each year and infect hundreds of millions more. Most of those infected are miles from a doctor and even farther from reliable medical equipment. This means that many go untreated, and many more may be misdiagnosed. A portable method for sampling blood in the field could literally save millions of lives. CellScope has the potential to do just that. Field personnel with little experience can take a blood sample, image it, then send the photo to a qualified medical professional miles away. Diagnosis by phone might be just what the doctor ordered.

The idea and first prototype for the CellScope came out of a class that Fletcher taught. He challenged his students to create a microscope that could be attached to a cell phone. Now, many of those same students are helping CellScope through the development phase. The current incarnation of the device is still a little bulky (as you can see in the video) but the ultimate goal is a few inches of length and less than one pound in weight.

While Malaria can be identified with standard light illumination, TB requires a fluorescent technique. So there is more than one model for the CellScope, including one with 460 nm wavelength light to help diagnosis TB. Other variations to match other diseases are currently in development. It’s unclear whether the final device will be a single tube with complicated means of switching lenses and illumination or if a field user will carry several different tubes to match different testing modalities.

While it may revolutionize medicine in remote locations, CellScope could also do the same domestically. Patients with chronic blood conditions or skin diseases could have a powerful new tool at their disposal. They may be able to stay at home and send images to doctors for diagnosis rather than make costly routine visits to hospitals.

It’s unclear when CellScope will hit the market and which market (domestic or international medicine) will be approached first. In fact, with all the prototyping and decision making that needs to be done, I’m worried that it may be several years or more before it is widely available.

I should point out that Prof. Dan Fletcher and his crew aren’t gadget gurus, they’re biophysicists, grad students and undergraduates. Which makes this whole story that much more impressive to me. Kudos to Dr. Fletcher for not just taking time out of research to help people, but for teaching his students to do the same. My own advisor always used to tell me that it wasn’t enough to just perform experiments, you had to give back to society as well. Science and social conscience, like remote diagnosis and microscopy, is a powerful combination. Batman would approve.

[photo credits: Fletcher Labs]