Is AI Really a Black Box? Neural Networks May Be Easier to Understand Than We Thought

The idea that the neural networks underpinning modern AI are “black boxes” almost impossible to understand has become received wisdom. But a provocative blog post from a researcher who played a key role in setting that narrative says these models may be far more interpretable than we thought.

The opacity of today’s leading AI models is becoming an increasingly pressing problem as the technology seeps into ever more areas of our lives. Recent incidents where agents from leading developers escaped their sandboxes to go on hacking sprees have only boosted the sense of urgency.

This has spurred interest in a research field known as mechanistic interpretability, which attempts to reverse engineer neural networks to learn what individual neurons and the circuits they create do. By detailing how these networks operate, researchers hope it’ll be easier to control and direct them.

Progress has been slow, however, and leading theories have suggested the way these networks learn leads to a tangled mess that is incredibly difficult to parse. Now though, Nick Cammarata, a researcher at AI startup Goodfire who helped lay the foundations of the field while at OpenAI, has published a 26,000-word blog post arguing that these theories are wrong and even the largest neural networks could be much simpler to read than expected.

“I think I have good news about the nature of neural networks,” Cammarata writes in the post. “Interpretability might be far more tractable, possibly in the near term, than we’ve assumed.”

The idea Cammarata is challenging is one he helped establish. In 2020, while studying a Google image-recognition model called InceptionV1, he and colleagues found neurons that seemed to respond to unrelated things—most famously one that fired on both cat faces and the fronts of cars.

The leading explanation was the concept of superposition, which suggested that when a network runs short of neurons, it starts lumping unrelated concepts together into the same neurons. And it specifically combines concepts that are so disconnected they’re unlikely to clash. This means that looking at an individual neuron tells you very little, because there’s no rhyme or reason behind what ideas it groups together.

But when Cammarata revisited the famous cat-car neuron, he realized both images shared a prominent V shape. Further tests showed the same neuron also responded to the corner of a pool table and the pointed nose of a Japanese bullet train.