Reading a Model's Mind
How researchers learned to pull a neural network's tangled thoughts apart into concepts you can name, and even turn up or down.
A model's thoughts are written in a code no human chose, smeared across millions of neurons; the surprise of the last two years is that the code can be read, one concept at a time.
Ask a language model anything and it answers fluently, confidently, and from somewhere you cannot see. Its knowledge is not stored in sentences or rules. It is stored as weights, billions of numbers fixed at the end of training, and at every step those weights turn the input into a long column of activations: numbers describing the model's internal state in the instant before it speaks. Understanding the model means understanding that column. For most of the field's history, nobody could.
The neuron that means everything
The natural place to look is a single neuron. Find the neuron for the Golden Gate Bridge, the neuron for sarcasm, the neuron for the Arabic language, and you have a map. But when researchers looked, individual neurons refused to mean one thing. A single neuron would fire for the bridge, and also for legal contracts, and also for a particular musical key, and the color burgundy. This is called polysemanticity, and it is not a flaw. It is a consequence of arithmetic.
A network has a fixed number of neurons but needs to represent far more concepts than that. So it cheats. It stores ideas not as single neurons but as patterns spread across many, and it overlaps those patterns, letting thousands of rarely co-occurring concepts share the same neurons the way a crowded language reuses the same few sounds for thousands of words. This overlap is called superposition. It is why the model is so capable, and why, read neuron by neuron, it is unreadable.
A neuron is not a word in the model's language. It is a letter, shared across thousands of words, meaningless until you see which word it is part of.
Pulling the threads apart
The breakthrough was to stop reading neurons and start reading directions. A concept in these models is not one neuron but a particular direction in the high-dimensional space of activations, a specific blend of many neurons firing together. The tool that recovers these directions is a sparse autoencoder. It is a small second network trained to do one narrow thing: take the model's tangled activations and re-express them using a much larger dictionary of possible features, under the rule that only a tiny handful may be active at once.
That sparsity constraint is the whole trick. Forced to explain every activation with only a few features at a time, the autoencoder cannot lean on superposition; it has to find the genuine, separable concepts. When Anthropic ran this on a production Claude model in 2024, a dictionary of millions of features re-expressed each activation using only a small number of features at any moment, and those features were clean. One responded to the Golden Gate Bridge in text, in any language, and in images of it. Monosemantic at last: one feature, one meaning.
Finding, naming, steering
A found feature can be named by watching what makes it fire and what it pushes the model to say. Features do not sit in isolation; they wire into circuits, chains where one feature feeds the next, so that a question about a country's capital lights a path from the country feature through a retrieval step to the city. Mapping those paths is how you move from a vocabulary of concepts to the model's actual reasoning, and it is where the hardest current work lives.
The proof that a feature is real, not a story we tell about numbers, is that you can turn it. Clamp the Golden Gate feature to an unnaturally high value and the model becomes fixated: ask it for a soup recipe and it will find its way to the bridge. That demonstration was playful, but the mechanism is not. The same handle that forces a model toward a bridge can be pointed at a feature for deception, or for a dangerous capability, and held down to see what the model does when that concept is artificially loud.
You have not truly understood a part of a mind until you can turn it up and watch the behavior bend in exactly the way your understanding predicted.
Why a discerning reader should care
These systems are being handed real authority while remaining, in the ordinary sense, unexplained. A model can be helpful for a thousand prompts and harmful on the next, and a black box gives you no way to know which features were active when it turned. By mid-2026 this had stopped being purely academic: interpretability began feeding into pre-deployment safety review, with researchers reading a model's internal features for signs of deception or hidden goals before release. It was among the first times that looking inside a model informed the decision to ship it.
It is worth being honest about the limits. Today's features are partial; circuits are mapped for narrow behaviors, not whole minds; an autoencoder leaves residue it cannot explain. This is early cartography, a few cleared paths through an enormous dark interior. But the direction is no longer in doubt. The code these models think in was supposed to be unreadable, and it is turning out, slowly and concept by concept, to be a code after all, which is the first requirement for ever trusting what we have built.