Picture this: you’re standing in front of a pile of glistening avocados at a grocery store. You snatch one up, put it in your cart, and wheel off dreaming of nachos and guacamole — a completely effortless interaction.
Now ask yourself: how did you know which avocados you could pick up without toppling the entire pile?
Not so simple, huh?
To answer the question, neuroscientists are taking a hint from game developers. Much like physical simulation engines that power some of the best computer games, we may have analogous neural networks that let us elegantly navigate the real world. Rather than getting mired in details, gaming engines take shortcuts to simulate actions in the gaming space in a way that looks good and makes sense, allowing players to react appropriately on the fly. The brain might work the same way.
In a paper published in the academic journal Proceedings of the National Academy of Sciences, a team from Johns Hopkins University, in collaboration with Harvard and MIT, describes the first glimpse of what might be a dedicated physical simulator spread across our brain.
All Hands on Deck
Say you need to suddenly duck from a ricocheting ball, dodge a tree branch, or swerve to avoid hitting a deer on the road.
Let’s slow those reactions down: you see the object — color, shape, material, shadow — sense its position and movement in three-dimensional space, plan out actions to take, then trigger your body to move precisely to avoid collision. That’s some serious multitasking in the brain.
“I see physical scene understanding as a highly integrative process that sources information from many brain areas working in concert,” says study lead author Dr. Jason Fischer to Gizmodo.
In other words, it’s actually not intuitive that the brain would have a chunk of tissue solely dedicated to physics simulation, one that lays dormant except in the case of emergencies. A hunt for such a region would be hopeless.
Instead, the authors asked a more astute question: against the backdrop of constant computation in the brain, are there any areas that specifically spring into high gear when brain meets physics?
Rev up the Engine
To weed out confounding background chatter, the researchers carefully crafted four experiments, each one building on top of the last.
In the first test, while lying in an fMRI machine, twelve participants watched videos of Jenga-like towers of yellow and blue blocks precipitously stacked on top of each other, ready to topple. The volunteers had to predict where the blocks would fall or whether there were more blue or yellow blocks. Unlike the first task that called upon physics intuition, the latter relied solely on color, acting as a control.
With this first sweep, the researchers narrowed their search to eleven brain regions that showed stronger activation when the volunteers were predicting landing as opposed to guessing color.
But are these “regions of interest” specific for physical intuition, or for simulating future events in general?
That’s where the second task came in. The volunteers watched video clips of two balls interacting, either physically — for example, bouncing off of each other — or in a social manner, such as one chasing the other like two kids playing. After a few seconds, one of the balls disappeared, and the volunteers had to guess where it would next move. Obviously, this requires simulation. But here’s the brilliant part: although the latter scenario also required prediction, it relied heavily on social reasoning rather than physics.
By picking out brain regions that preferentially fired up in the physics game and cross-referencing with the results from the first task, the researchers were able to narrow down the candidate regions to only five. They validated these areas in the third task, where volunteers passively watched videos with varying degrees of physics-centric material, such as a still face, a falling vase, or colliding cars, while having their brains activity scanned by an fMRI.
The result: the more physical content in the videos, the more those five regions activated, even when the volunteers weren’t actively trying to predict what would happen next.
Finally, the researchers asked how specialized these brain areas are for physics simulation. As it turns out, not so much: some of the same regions fired up while the volunteers were engaged in difficult memory-taxing problems that had nothing to do with physics.
So why this generalization?
As it turns out, it may have something to do with the difficulty of predicting how the world behaves. Previous research tells us that faced with complex problems, the brain usually calls upon a series of interconnected brain regions dubbed the multiple-demand network. True to its name, activation of this network helps with demanding problem solving, such as planning your next movement and using intricate new tools.
The researchers found that the brain’s physics engine — this group of five regions — significantly overlaps with the multiple-demand network. Predicting dynamic movement isn’t the engine’s only purpose, but it sure is one of its major jobs.
All Systems Go
Does it make sense to talk about a “physics engine” when this brain region also does other things?
The researchers pointed to graphic processing units (GPUs) as an analogy.
“The highly parallel architecture of GPUs was originally motivated by the demands of graphics-intensive computing applications,” explained the authors in their paper, “but GPUs have since become indispensable for other applications, such as computer vision, deep neural network training, and indeed, real-time approximate physics simulation in computer games.”
GPUs are active during all those tasks, meaning that it’s a physics engine, but also a graphics engine, a computer vision engine, and so on. In a way, the brain network identified in this study acts like a biological GPU — it supports physics inference but is built into a larger network responsible for other difficult tasks, such as action planning.
Why would inference and planning be so intricately linked in the brain?
“We believe this might be because infants learn physics models of the world as they hone their motor skills, handling objects to learn how they behave. Also, to reach out and grab something in the right place with the right amount of force, we need real-time physical understanding,” explains Fischer.
This type of understanding breaks down in a disease called apraxia, where people have difficulty performing certain movements due to brain damage.
“Many cases of apraxia result from damage to the same brain areas that we have identified as important for movement,” says Fischer to Singularity Hub. Future studies should test whether temporarily shutting down the brain’s physics engine disrupts physical inference and mimics the symptoms of apraxia. If so, the faulty physics engines may be a good place to start fixing.
By studying how the regions of the brain’s physics engine communicate with one another, we may even be able to build robots that run on better physics models.
“In this study we only tested a small subset of all possible types of physics reasoning,” wrote the authors. For example, would our brains respond to interacting liquids — two waves crashing into each other — the same way they respond to two solid objects smashing together?
If the brain’s physics engine did behave like a gaming physics simulator, it would be specialized for only a small subset of conditions — liquids or solids, for example. That means we may have more types of physics engines buried inside our vast neural computations.
By teasing apart how each one works, we may be able to power robots with multiple physics engines constantly running like a video game. “If we are able to build robots that quickly and efficiently simulate the outcomes of physical scenarios, then just like humans, they can have a sense that an event will happen before it actually does,” says Fischer. They’ll be able to interact with their changing environments with human-like grace.
Physics simulation is truly among the most important aspects of cognition for survival, says Fischer. Thanks to this study, we finally have an inkling of how it works.
Banner image credit: Shutterstock
