Combining fMRI and a new computational technique researchers can scan a person's visual cortex and recreate roughly the movie they're watching.

Scientists at the University of California, Berkeley can now read your mind and project the images you think of onto a video screen. What sounds like science fiction may one day allow clinicians to “see” what a person in a coma is thinking, allow sleep researchers to watch dreams, or improve brain-machine interfaces that control robotic limbs.

Led by neuroscientist Jack Gallant, the researchers combined brain scanning technology with sophisticated computation to read their subjects’ visual minds. Functional magnetic resonance imaging (fMRI), which measures blood flow in the brain, was performed while three different subjects watched movie trailers from the Apple QuickTime gallery and YouTube. The scans focused on the visual cortex, the part of the brain that processes visual information. The first time the trailers were shown the computer simply recorded the corresponding brain activity. In this way it “learned” what the screen images look like in terms of brain activity patterns. The subjects were then shown a different set of movie trailers. This time the computer had to read the brain activity and try to “guess” what it was seeing. The computer then used a reconstruction algorithm of the 100 trailers most likely being viewed by the subject and merged them into a single display. The following video shows a subjects’ trailer alongside the computer’s best guess amalgam. I think you’ll agree that the match is pretty remarkable.

Because the computer is choosing from a group of already-made movie clips it is not actually reading brain activity and producing an image de novo – which explains the random numbers and other seemingly nonsensical imagery in the videos. That amazing feat still remains for future researchers to achieve. But using fMRI to accurately encode realtime motion pictures is already a breakthrough.

Although fMRI is routinely used to measure brain activity, it isn’t actually capable of detecting the activity of neurons directly. What it does detect are changes in blood flow within the brain. Because active neurons need energy, blood flow will increase in areas of high neuronal activity. fMRI has been a useful tool to neuroscientists, but one drawback is its low time resolution. Changes in blood flow can’t keep up with the much faster changes in the activity of neurons. This is why past attempts to produce images from brain activity have been mostly focused on still images. The major advance in the current study was to create a computational model that translates the fMRI signals into neuronal activity, rather than simply measuring the changes in blood flow.

Dr. Gallant thinks his advancements could lead to visualizing another, more mysterious type of movie. “This is a major leap toward reconstructing internal imagery,” he said in a press release. “We are opening a window into the movies in our minds.”

The authors of the report hope that accurately modeling dynamic thought processes could be used one day to help diagnose psychiatric disorders or provide the basis for brain-machine interfaces. “Our natural experience is like watching a movie,” Shinji Nishimoto, the study’s lead author, said in the press release. “In order for this technology to have wide applicability, we must understand how the brain processes these dynamic visual experiences.”

Earlier this year researchers at Washington University in St. Louis implanted a net of electrodes on the surface of patients’ brains that detected neuronal activity and transmitted the signals to a computer. With this brain-computer interface the patients were able to manipulate a cursor on the computer screen using just their minds. While the participants in the current study don’t perform any active manipulations, the study demonstrates that fMRI signals might do the trick. If so, it would be a major advance for mind-controlled prosthetic limbs. Because fRMI is non-invasive – as opposed to a sheet of electrodes across your brain – it can harmlessly be used to read signals from any part of the brain.

Of course, until they get a lot smaller, walking around with an MRI machine strapped to your head is not feasible. Nor is casually toting a magnet so incredibly powerful it turns paperclips into projectiles.

Regardless, combining the time resolution necessary to reproduce movies with fMRI’s global scanning capabilities, the technique could be soon be used to scan the more esoteric happenings inside a person’s brain such as their daydreams, emotional state, or intentions.

I can just see it now.


video: Gallant Lab

Peter Murray was born in Boston in 1973. He earned a PhD in neuroscience at the University of Maryland, Baltimore studying gene expression in the neocortex. Following his dissertation work he spent three years as a post-doctoral fellow at the same university studying brain mechanisms of pain and motor control. He completed a collection of short stories in 2010 and has been writing for Singularity Hub since March 2011.