Reverse-Engineering the AI Mind
How Mechanistic Interpretability Went From Manual Circuit-Mapping to Dreaming AI Agents
Glass box, not a wall—
gradients grow strange gardens.
We learn to translate.
With every article and podcast episode, we provide comprehensive study materials: References, Executive Summary, Briefing Document, Quiz, Essay Questions, Glossary, Timeline, Cast, FAQ, Table of Contents, Index, Polls, 3k Image, Fact Check, Comic and
Street Art at the very bottom of the page.
Soundbite
Trailer
Essay
We Built Minds We Can’t Read Yet — And That’s Not the Whole Story
Here’s an uncomfortable fact you’ve probably made peace with without noticing: the AI system that helped you draft an email this morning, the one that summarized a contract or wrote a snippet of code, was built by people who cannot tell you how it actually did any of that. Not in the way an engineer can explain a bridge or a mechanic can explain an engine. We wrote the training process. We did not write the mind that emerged from it.
This is the paradox sitting at the centre of a young science called mechanistic interpretability, and it’s worth sitting with for a moment, because the instinct is to assume someone, somewhere, understands the machine. Someone does not. Not fully. Not yet.
The confusion, once you unpack it, is almost embarrassingly simple. We wrote the rules for learning — the back propagation algorithm, the loss function, the gradient descent mechanics that tell a neural network how to adjust itself when it’s wrong. But we did not write the thing that learned. What emerges from millions of rounds of self-correction is not code in any traditional sense. It’s a colossal structure of weighted connections, arrived at the way a coral reef arrives at its shape: not designed, but grown, generation by generation, under pressure. The training code is the evolutionary force. The resulting network is the organism that survived it.
So when researchers talk about “reverse engineering” a language model, they don’t mean debugging a program they misplaced the manual for. They mean doing something closer to biology: dissecting a living structure to understand what it became, not what it was told to be.
The Optimists, and What They Found
For a while, this looked tractable, even elegant. In 2021, a team including Neel Nanda, Chris Olah, and colleagues published a paper that gave researchers a genuine mathematical vocabulary for describing what happens inside a transformer — not as one incomprehensible blob, but as a series of interacting circuits, each doing something specific. The centrepiece discovery was something called an “induction head”: a two-part mechanism, spread across different layers, that lets a model notice a pattern earlier in a conversation and apply it later. Type a made-up variable name once, refer back to it fifty lines later, and the model gets it right — not because it memorized your specific words during training, but because it learned, in real time, to bind a new concept to its context and retrieve it. That’s not a party trick. That capability, the paper showed, is close to the mathematical seed of what makes these systems useful at all.
It was a genuinely thrilling result, and it set off a wave of optimism. If induction heads could be found and mapped, why not everything else? Neel Nanda — a Cambridge-trained mathematician who’d studied under Olah at Anthropic before eventually leading DeepMind’s interpretability team in his mid-twenties — became something like a rallying figure. He wrote a list of 200 open problems for the field to chew on. He built free tools so outside researchers could peer into a model’s neurons themselves. The underlying hope was called the “universality hypothesis”: the idea that neural networks, regardless of their specific training, tend to converge on the same handful of clever solutions — much the way eyes evolved independently, more than once, because an eye is simply what works for processing light. If that held true, mapping one model’s mind would mean mapping something close to a universal law of artificial cognition.
And for a while, the wins kept coming. Researchers isolated the exact circuit a model uses to correctly identify who received the milk bottle in the sentence “John gave a bottle of milk to—” (it says Mary, and researchers could show precisely which attention heads suppressed “John” and boosted “Mary” to make that happen). It felt like watching grammar gears turn inside a machine that was, for once, being honest with us.
Then the Ground Shifted
Here’s the part of the story that a lot of early coverage skipped: the neat successes were, in a real sense, cherry-picked — not through dishonesty, but through necessity. Critics started calling it “streetlight interpretability,” after the old joke about the drunk searching for his keys under the lamppost, not because he dropped them there, but because that’s where he can see. Researchers were mapping grammar circuits because grammar is unusually clean and measurable. The vast, messy remainder of the model’s parameters sat largely unexamined.
The deeper problem had a name: superposition. A model has a fixed, limited number of dimensions to work with at each layer, but it needs to represent vastly more concepts than it has room for — sometimes a hundred times more. The solution networks discovered on their own is genuinely beautiful math and genuinely bad news for anyone hoping for a tidy map: instead of one neuron per concept, models pack many concepts into overlapping, almost-but-not-quite-perpendicular directions through that space, betting that the concepts rarely need to activate at the same time. It’s like a traveler stuffing a swimsuit inside a snow boot to save luggage space — brilliant, provided you never need both on the same trip. But it means a single neuron can fire for completely unrelated things — the 2021 papers found one that responded to both pictures of dice and pictures of poets — and any researcher who confidently labels a neuron based on one data set is likely fooling themselves. This became known, aptly, as the interpretability illusion, and it is disarmingly easy to fall into even when you know it’s there.
By the mid-2020s, Nanda himself said the quiet part out loud: the most ambitious version of this dream — fully decompiling a frontier model into human-readable logic — is probably dead. Not because the researchers weren’t clever enough, but because the scale simply doesn’t cooperate. It took a team of researchers months to hand-map one grammar circuit in a modest 117-million-parameter model. Frontier models today have parameters numbering in the hundreds of billions. The math doesn’t close.
A Quieter, More Honest Goal
What’s genuinely encouraging is what happened next: the field didn’t fold—it adjusted its ambitions into something more useful. Enter sparse autoencoders: tools that take a model’s dense, overlapping activity and deliberately expand it into a much wider space, penalizing overlap until each concept gets something closer to its own dedicated slot. DeepMind’s team, under Nanda, released hundreds of these tools openly, giving the whole research community shared instruments rather than isolated observations. The resulting maps, called attribution graphs, are technically legible — but they’re now so enormous that no human can trace them by hand either. So researchers are doing something almost recursive: building AI agents to interpret other AI systems. A 2024 MIT project called MAIA showed a vision-language model could design its own experiments to figure out what a neuron responds to. A 2026 system called Circuit Oracle goes further, dispatching swarms of sub-agents to trace causal pathways backward through a model’s reasoning and flag when a prediction is being driven by something spurious, or hidden, or concerning.
The honest goalpost has moved from understanding everything to finding the dangerous parts — the way an MRI isn’t built to map every healthy cell in your body, just to flag the tumour. That’s a smaller ambition than the field started with. It’s also, arguably, the one that actually matters for safety.
There’s a real, unresolved tension worth naming plainly: we’re now using uninterpretable systems to explain other uninterpretable systems, and these interpreting agents inherit the same blind spots — confirmation bias, tunnel vision, a tendency to declare victory the moment a hypothesis gets one supporting data point. The researchers building these tools know this. Which is why, even as the pipeline gets faster and more automated — down to genuinely striking recent work letting agents “sleep” between sessions to consolidate what they’ve learned, while carefully preserving an unaltered, permanent record of every raw experiment underneath — the human scientist hasn’t been written out of the process. They’ve been repositioned: less as a cartographer labouring over every inch of the map, more as the principal investigator asking hard questions of a very fast, very tireless, occasionally overconfident research assistant.
That’s not a failure story. It’s closer to what real science usually looks like — messier than the early optimism promised, more collaborative than the lone-genius myth suggests, and still, genuinely, moving. We may never get the complete, human-readable source code for a mind that grew rather than was written. But we are, for the first time, building the instruments to ask it better questions — and, crucially, to catch it when it’s lying to us. Given where this started five years ago, that’s not a small thing. It might be the thing that actually matters.
Link References
A Mathematical Framework for Transformer Circuits
In-context Learning and Induction Heads
Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
Progress measures for grokking via mechanistic interpretability
A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations
Refusal in Language Models Is Mediated by a Single Direction
Actually, Othello-GPT Has A Linear Emergent World Representation
A Multimodal Automated Interpretability Agent
Circuit Oracle: Automating Attribution Graph Analysis via Natural-Language Queries
200 Concrete Open Problems in Mechanistic Interpretability
A Comprehensive Mechanistic Interpretability Explainer & Glossary
Andrej Karpathy’s LLM Wiki gist (llm-wiki.md)
Researchers
Neel Nanda Academic/Institutional Affiliations: Google DeepMind (Senior Research Scientist and Mechanistic Interpretability Team Lead), formerly Anthropic and Trinity College, Cambridge.
Andrej Karpathy Academic/Institutional Affiliations: Stanford University, formerly Tesla and OpenAI.
Chris Olah Academic/Institutional Affiliations: Anthropic.
Nelson Elhage Academic/Institutional Affiliations: Anthropic.
Catherine Olsson Academic/Institutional Affiliations: Anthropic.
Tom Henighan Academic/Institutional Affiliations: Anthropic.
Ben Mann Academic/Institutional Affiliations: Anthropic.
Hong Kiat Tan Academic/Institutional Affiliations: University of California, Los Angeles (UCLA).
Shariar Kabir Academic/Institutional Affiliations: Not explicitly detailed in the sources.
Swastik Agrawal Academic/Institutional Affiliations: University of Maryland.
Sai Chereddy Academic/Institutional Affiliations: University of Maryland.
Sriram Balasubramanian Academic/Institutional Affiliations: University of Maryland.
Tamar Rott Shaham Academic/Institutional Affiliations: MIT CSAIL.
Sarah Schwettmann Academic/Institutional Affiliations: MIT CSAIL.
Franklin Wang Academic/Institutional Affiliations: MIT CSAIL.
Achyuta Rajaram Academic/Institutional Affiliations: MIT CSAIL.
Evan Hernandez Academic/Institutional Affiliations: MIT CSAIL.
Jacob Andreas Academic/Institutional Affiliations: MIT CSAIL.
19. Antonio Torralba Academic/Institutional Affiliations: MIT CSAIL
Episode Links
🍎 Apple
🎵 Spotify
▶️ YouTube
🎧 Listen
Or subscribe on your favourite podcast provider!
📖 Read
📻 Available for Broadcast on PRX
📻 PRX Series:
Grown, Not Written: Two Ways of Trying to Read a Mind We Didn’t Design
Other Links to Heliox Podcast
YouTube
Substack
PRX ( Public Radio Exchange)
Podcast Providers
Spotify
Apple Podcasts
Patreon
FaceBook Group
STUDY MATERIALS
Executive Summary
Mechanistic Interpretability (MI) is the study of reverse-engineering neural networks from learned weights into human-interpretable algorithms. The field operates on the fundamental hypothesis that models represent “features”—properties of inputs—as specific directions in high-dimensional activation space. These features are connected by “circuits,” which are computational subgraphs consisting of weights and non-linearities that perform specific tasks.
Key findings indicate that transformers utilize a “residual stream” as a shared communication channel between layers. Attention heads function primarily as information routers, while Multi-Layer Perceptron (MLP) layers perform conceptual processing, including Boolean operations and thresholding. A significant challenge in MI is “superposition,” where models represent more features than they have dimensions, leading to “polysemantic” neurons that respond to multiple unrelated concepts. Modern methodologies, such as activation patching and direct logit attribution, allow researchers to localize and verify these internal mechanisms with increasing precision.
I. Core Concepts of Mechanistic Interpretability
Features and Circuits
Feature: A property of an input that is internally represented by the model. Features are the fundamental building blocks of model computation. Examples include “this text is Python code” or “this token is a number referring to a group of people.”
Circuit: The subset of a model’s weights and non-linearities that map earlier features to later features to perform a specific computation.
Universality Hypothesis: The idea that different models trained on similar tasks will converge on similar circuits and internal representations.
The Linear Representation Hypothesis
The prevailing framework in MI is that features correspond to directions in activation space. This allows models to use linear algebra (projections and additions) to combine and manipulate information.
Privileged Basis: A meaningful basis where coordinates have intrinsic meaning (e.g., neurons following a ReLU non-linearity).
Bottleneck Dimensions: Low-dimensional spaces (like the residual stream or attention keys/queries) that lack a privileged basis, making them harder to interpret directly.
II. Transformer Architecture and Information Flow
The Residual Stream
The residual stream is the central “shared memory” of the transformer. Every layer reads from it and writes back to it via addition. This structure allows information to persist across many layers and enables “skip connections” where layer n might only be read by layer n+5.
Attention Mechanisms: Information Routing
Attention heads move information between different token positions in a sequence. This is decomposed into two circuits:
QK-Circuit (Query-Key): Determines where to move information from and to. It acts as a bilinear form identifying relevant source tokens for a destination token.
OV-Circuit (Output-Value): Determines what information is actually moved once the connection is established.
MLP Layers: Conceptual Processing
MLP layers represent approximately two-thirds of a transformer’s parameters but are less understood than attention. They are theorized to perform the following operations:
Boolean AND: Activating only if multiple features (e.g., “ice” AND “cream”) are present.
Thresholding/Sharpening: Mapping a continuous evidence signal into a discrete feature (e.g., “is at least 3 occurrences of X”).
Lookup Tables: Mapping specific input directions to output directions (e.g., “Bill Gates” → “founded Microsoft”).
IV. Superposition and Polysemanticity
Models often encounter the “curse of dimensionality,” where they must represent more features than they have available dimensions.
Definitions
Superposition: Representing M features in N dimensions where M>N. This is a form of lossy compression.
Polysemanticity: A single neuron responding to multiple, unrelated features. This is a direct consequence of superposition.
Monosemanticity: A neuron responding to a single, consistent feature (rare in standard models).
Drivers of Superposition
Superposition is incentivized by:
High Sparsity: Features that rarely appear in the input.
High Importance: Features that significantly reduce model loss when correctly identified.
Solutions: The SoLU Activation Function
The Softmax Linear Unit (SoLU) is an alternative activation function designed to inhibit superposition by encouraging lateral inhibition. It forces neurons to compete via a softmax, making them more likely to be monosemantic and interpretable compared to standard GELU activations.
V. Key Circuit Case Studies
1. Induction Circuits
Induction heads are a universal mechanism for “in-context learning,” allowing models to continue repeated subsequences (the [A][B]...[A]→[B] pattern).
Mechanism: A two-head circuit. A “previous token head” in an early layer writes information about the preceding token into the residual stream. An “induction head” in a later layer looks for this information to predict the next token.
Significance: These heads appear suddenly during a “phase transition” in training, coinciding with a visible improvement in the model’s loss curve.
2. Indirect Object Identification (IOI)
A complex circuit in GPT-2 Small that completes sentences like “John and Mary went to the store, John gave the bag to [Mary].”
Heads Involved: Duplicate token heads (identify repeats), S-Inhibition heads (suppress the subject’s name), and Name Mover heads (promote the indirect object).
Redundancy: The circuit contains “Backup Name Mover Heads” that activate only if the primary heads are ablated, suggesting a robust internal architecture.
VI. Analytical Methodologies
To identify and verify these mechanisms, researchers use several rigorous techniques:
Ablation (Knockout): Setting a component’s output to zero (or its mean) to see if performance on a specific task degrades.
Activation Patching: Replacing a specific activation in a “corrupted” run with one from a “clean” run to localize where a specific computation occurs.
Direct Logit Attribution: Decomposing the final residual stream to see which specific head or neuron contributes most to the correct next-token logit.
Causal Scrubbing: A rigorous algorithm for testing whether a proposed computational subgraph (circuit) is sufficient to explain a model’s behavior.
Probing: Training a linear classifier on internal activations to see if a specific feature is present in the residual stream.
VII. Pitfalls and Challenges
Interpretability Illusion: A neuron may appear monosemantic on one dataset (e.g., Wikipedia) but respond to completely different features on another (e.g., Python code).
Hypothesis Confirmation Bias: It is easy to find a pattern in a neuron’s activation and assume it is the “true” feature. Rigorous testing requires attempting to falsify the hypothesis by feeding the model counter-examples.
Neuron Splitting: As models scale, a single general feature (e.g., “hexadecimal”) may split into many specific features (e.g., “the digit 3 in hexadecimal”) across multiple neurons.
Causal Attention Constraints: Because GPT-style models are causal, a neuron’s activation at position k cannot be influenced by any token at position k+1 or later. Text appearing after the activation point in dataset examples is irrelevant to the model’s computation.
Quiz & Answer Key
Instructions: Answer the following questions in 2-3 sentences based on the provided research context.
What is the primary objective of Mechanistic Interpretability?
How does the “features as directions” hypothesis describe model internal activations?
What role do MLP layers play in a Transformer, and why are they considered difficult to understand?
Define “superposition” and explain why a model might employ it.
What is the difference between a “monosemantic” and a “polysemantic” neuron?
Explain the “residual stream” and its function as “shared bandwidth” within a Transformer.
What are “induction heads,” and why are they significant in the study of in-context learning?
Describe the “SoLU” activation function and its intended effect on neuron interpretability.
What is “activation patching,” and how does it help in identifying model circuits?
Distinguish between “sensory neurons” and “motor neurons” within the context of generative language models.
Quiz Answer Key
What is the primary objective of Mechanistic Interpretability? Mechanistic Interpretability is the field of reverse engineering neural networks from their learned weights into human-interpretable algorithms. It aims to transform “compiled” program binaries (the model weights) back into understandable “source code” or legible computational subgraphs.
How does the “features as directions” hypothesis describe model internal activations? This hypothesis suggests that features are represented as specific directions in a model’s high-dimensional activation space. Layers can access these features by projecting onto these directions, allowing multiple independent features to be stored and manipulated via linear algebra.
What role do MLP layers play in a Transformer, and why are they considered difficult to understand? MLP layers represent approximately two-thirds of a Transformer’s parameters and are responsible for processing information at specific token positions. They are difficult to understand because their activations are often messy, features may not align perfectly with individual neurons, and issues like polysemanticity make them hard to reverse engineer.
Define “superposition” and explain why a model might employ it. Superposition occurs when a model represents more than n features in an n-dimensional activation space, effectively using lossy compression. Models employ this to represent a vast number of sparse, niche features that are useful for lowering loss, even if it introduces some interference or noise.
What is the difference between a “monosemantic” and a “polysemantic” neuron? A monosemantic neuron corresponds to a single interpretable feature, making its function clear across its activation range. Conversely, a polysemantic neuron responds to multiple unrelated features, which complicates interpretation as the neuron’s activation is shared by different concepts.
Explain the “residual stream” and its function as “shared bandwidth” within a Transformer. The residual stream is the central object of a Transformer, acting as a shared memory where all layers read from and write to. It is considered “shared bandwidth” because it is the only mechanism for moving information between layers, requiring the model to manage which features are stored there at any given time.
What are “induction heads,” and why are they significant in the study of in-context learning? Induction heads are specialized attention heads that detect and continue repeated subsequences, such as predicting “Bond” after “James” if “James Bond” appeared earlier. They are significant because they appear to be universal across models and are a primary driver of the model’s ability to perform in-context and few-shot learning.
Describe the “SoLU” activation function and its intended effect on neuron interpretability. The Softmax Linear Unit (SoLU) is an activation function designed to make neurons more monosemantic by using lateral inhibition and sparsity. It encourages the model to represent features using single neurons rather than diffuse directions, thereby making the standard basis more interpretable.
What is “activation patching,” and how does it help in identifying model circuits? Activation patching is a causal intervention technique where an activation from a “clean” model run is swapped into a “corrupted” run to see if it restores the correct output. This allows researchers to localize exactly which components (like specific heads or neurons) are responsible for a particular model behavior or circuit.
Distinguish between “sensory neurons” and “motor neurons” within the context of generative language models. Sensory neurons are found in early layers and convert raw token data into useful internal representations, such as merging tokens or identifying context. Motor neurons are found in later layers and convert conceptual understandings back into specific output actions, such as predicting the next token in a multi-token word.
Essay Questions
Instructions: These questions are designed to stimulate deep analysis and synthesis of the source material. No answers are provided.
The Neuron vs. Direction Debate: Discuss the evidence for and against the idea that neurons are the fundamental units of meaning in language models. Compare the “features as neurons” hypothesis with the “features as directions” hypothesis, citing the challenges posed by superposition.
The Impact of Activation Functions: Analyze how the choice between GELU and SoLU affects the interpretability of a model. In your essay, address the concept of “smuggling” features through LayerNorm and how lateral inhibition influences the sparsity of representations.
Circuit Analysis and the IOI Task: Using the Indirect Object Identification (IOI) circuit as a case study, explain the roles of “Name Mover,” “S-Inhibition,” and “Duplicate Token” heads. How does this circuit illustrate the concept of “information routing” in Transformers?
Scale and Emergence in MI: Evaluate how model size (scaling) influences the types of features learned by a Transformer. Discuss the transition from simple syntax/bigram neurons in small models to “sophisticated” conceptual neurons in larger models like GPT-2 XL.
The Utility of Toy Models: Critique the use of one-layer or attention-only toy models in Mechanistic Interpretability research. To what extent do the discoveries made in these simplified environments (such as induction heads) generalize to massive, multi-layer generative models?
Glossary of Key Terms
Ablation
The process of removing or “knocking out” a component of a network (setting it to zero or a mean value) to observe its impact on performance.
Activation Patching
A causal intervention technique used to identify which internal activations are necessary for a specific model output.
Attention Head
A component of the attention layer that moves information between different token positions in a sequence.
BOS (Beginning of Sequence)
A special token at the start of a context that gives attention heads a “resting position” to attend to when they are not active.
Circuit
A computational subgraph of a model consisting of a set of features and the weights/non-linearities that connect them to perform a task.
Causal Scrubbing
A rigorous method for testing whether a specific computational subgraph (circuit) explains a model’s behavior.
Direct Logit Attribution
A technique that measures the direct contribution of a specific model component’s output to the final logit of the correct next token.
Feature
A property of an input that is internally represented by the model (e.g., “this text is Python code” or “this is a proper noun”).
GELU
Gaussian Error Linear Unit; a smooth, non-monotonic activation function commonly used in modern Transformers.
Induction Head
A specific type of attention head that completes patterns by looking for previous occurrences of the current token and predicting the token that followed it.
Lateral Inhibition
A property where the activation of one neuron suppresses the activation of others, often used in SoLU to encourage monosemanticity.
Logit Lens
An interpretability technique that applies the model’s unembedding to intermediate residual stream layers to see the model’s current “best guess.”
MLP (Multi-Layer Perceptron)
Layers in a Transformer that apply non-linear processing to information at a specific position; they contain the majority of model parameters.
Monosemanticity
The state in which a neuron or direction corresponds to exactly one interpretable feature.
Neuroscope
A tool that displays the text examples that most strongly activate each neuron in various language models.
OV-Circuit
The “Output-Value” circuit of an attention head, which determines what information is moved once a source token is selected.
Polysemanticity
The state in which a single neuron responds to multiple, unrelated features.
Privileged Basis
A specific coordinate system (like individual neurons) that has a unique meaning due to non-linear operations like ReLU or GELU.
QK-Circuit
The “Query-Key” circuit of an attention head, which determines where the head attends (which source tokens it looks at).
Residual Stream
The sum of all previous layer outputs; the central vector space that layers read from and write to in a Transformer.
SoLU
Softmax Linear Unit; an activation function designed to increase neuron interpretability by applying a softmax across the neuron dimension.
Superposition
The phenomenon where a model represents more features than it has dimensions by using non-orthogonal directions in activation space.
TransformerLens
A software library designed for editing and accessing the internal activations of Transformer models for interpretability research.
Unembedding
The final linear map that converts the residual stream into logits, representing the probability distribution for the next token.
Gemini Notebook can be inaccurate; please double check its responses.
Cast of Characters
1. Prologue: The Search for Meaning in the MLP Black Box
In the mission to achieve AI Alignment, Multi-Layer Perceptrons (MLPs) represent our most critical strategic frontier. While MLPs constitute two-thirds of a transformer’s parameters, they have long been dismissed as inscrutable “black boxes.” Mechanistic interpretability challenges this opacity by positing the “features as directions” hypothesis: that the model represents concepts—from simple syntax to abstract semantics—as specific directions in its high-dimensional activation space. However, we face a fundamental tension between the “Standard Basis” and “Superposition.” If features aligned perfectly with individual neurons (monosemanticity), our task would be trivial. Instead, models utilize superposition to compress a vast number of features into a limited set of dimensions, leading to polysemanticity.
As researchers, we must recognize that if features correspond to arbitrary directions rather than the standard basis, staring at individual neurons in tools like Neuroscope offers only a glimpse of a much deeper, diffuse iceberg. The strategic imperative is to move beyond mere observation and toward the rigorous reverse-engineering of the “computational society” inside the model. To understand this society, we must first categorize the specialized characters that inhabit its layers.
2. The Sensory Guild: Early-Layer De-tokenization Neurons
The initial stages of a transformer function as the “Sensory Guild,” a specialized cadre responsible for converting raw, fragmented inputs into a semantically useful format. This is a strategic necessity because Byte-Pair Encoding (BPE)—the standard method for tokenization—is fundamentally arbitrary and often breaks coherent words into nonsensical fragments. Sensory neurons resolve this bottleneck by performing “de-tokenization,” merging pieces like “Donald | Trump” or “social | security” back into single conceptual units.
A hallmark of this guild is the presence of “equivariance” or neuron families. For example, researchers have identified families of neurons that detect the token “die” but distinguish its semantic role across German, Dutch, or English contexts. By resolving these ambiguities early, the Sensory Guild ensures that the residual stream—the model’s additive linear channel—is populated with stable representations. This foundational work allows downstream layers to move past raw syntax and begin the complex work of conceptual reasoning.
3. The Conceptual Strategists: Middle-Layer Sophisticated Neurons
Once the sensory data is stabilized, the “Conceptual Strategists” in the middle layers take over. This is the model’s “engine room,” where information moves beyond local syntax into high-level semantic abstraction. These layers are home to “Context Neurons,” which activate for broad environments like “is_base64_code” or “is_python_comment,” and “Sophisticated Neurons” that represent nuanced groupings, such as numbers implicitly referring to groups of people.
The strategic importance of MLPs here lies in their non-linearities (GELU or SoLU). While attention heads are excellent at routing information, MLPs are the only site where the model can perform “Boolean AND” logic or “Lookup Table” operations. By utilizing thresholding and sharpening, the model can take a continuous range of evidence (Feature A AND Feature B) and map it to a discrete, higher-order concept (Feature C). For instance, an MLP can “sharpen” various cues to decide definitively whether a block of text is Python code, a process of thresholding that attention-only architectures cannot easily replicate. This sharpening is what transforms a “noisy sum” of evidence into a clear conceptual signal.
4. The Motor Corps: Late-Layer Re-tokenization Neurons
As the signal reaches the final layers, the model transitions from “thought” back to “action.” This is the domain of the “Motor Corps,” or re-tokenization neurons. While the Sensory Guild deconstructs the input, the Motor Corps faces a “high-loss” bottleneck: it must translate a finalized conceptual decision back into the specific BPE tokens required by the objective function.
Consider the concept “nappies.” Once the Strategists have concluded this is the next word, the Motor Corps must fire precisely to output the sequence “n|app|ies.” These neurons are particularly active in the middle of multi-token words, ensuring the correct continuation of the sequence. We evaluate their impact through “Logit Attribution,” measuring how these characters directly boost the probability of the correct next token. Without the Motor Corps, the model’s conceptual brilliance would fail to manifest as predictive accuracy.
5. The Infrastructure Crew: Maintenance and Management Neurons
Not all neurons represent external features; a massive “Infrastructure Crew” is required to manage the model’s internal environment. The residual stream functions as an additive linear channel, and without active management, it would quickly become a chaotic, noisy sum of every preceding layer’s output.
The Infrastructure Crew performs two vital roles:
Memory Management Neurons: Identified by high negative cosine similarity between input and output weights, these neurons “delete” information from the residual stream once it has been processed by a previous layer, clearing bandwidth for future computations.
Signal Boosting Neurons: Because the residual stream’s vector norm grows significantly across layers, original signals like positional embeddings can become “diluted.” Signal boosting neurons (high positive cosine similarity) promote these essential features to ensure they remain a significant fraction of the total signal.
Additionally, this crew handles “Superposition Cleanup,” mitigating the interference that occurs when the model compresses too many features into few dimensions. They ensure the Conceptual Strategists have a clear, high-fidelity signal to work with.
6. The Specialized Envoys: Induction and Duplicate Token Heads
The neuron characters at each position are linked by “Specialized Envoys”—Attention Heads that route information across the sequence. To understand these envoys, we must distinguish between the QK-Circuit (the “routing” mechanism that determines where to look) and the OV-Circuit (the “moving” mechanism that determines what to copy).
The most sophisticated envoy is the Induction Circuit, a two-head system that enables in-context learning. It begins with the Previous Token Head, which writes the feature “the token before me is X” into the residual stream via K-Composition. The Induction Head then uses its QK-Circuit to search for this specific feature at earlier positions. When it find a match, its OV-Circuit “moves” the subsequent token (the “target”) to the current position, allowing the model to complete repeated patterns.
This routing logic is further evidenced in the Indirect Object Identification (IOI) circuit. Here, a “check-and-balance” system emerges: S-Inhibition Heads identify repeated subjects and inhibit Name Mover Heads from attending to the wrong target. The existence of Backup Name Movers—heads that only activate if the primary movers are ablated—reveals a level of internal redundancy and adversarial-like robustness that suggests the model functions less like a single program and more like a resilient computational society.
7. Conclusion: The Roadmap for Reverse Engineering
The categorization of these characters is the first step toward Enumerative Safety—the goal of identifying every feature within a model to ensure it is aligned with human values. We are moving from a period of “staring at neurons” to a new era of “rigorously reverse-engineering” the circuits they form.
By treating the transformer as a complex, multi-agent society rather than a monolith, we can begin to decode the internal logic that governs its behavior. The strategic imperative for Mechanistic Interpretability is to master this "reverse-engineering" process, ensuring the systems we build are not just capable, but fundamentally verifiable and safe.
Timeline of Main Events
1. The Strategic Landscape of Mechanistic Interpretability (MI)
Mechanistic Interpretability (MI) is the rigorous process of reverse engineering neural networks from their learned weights back into human-interpretable algorithms. It is strategically analogous to reverting a compiled binary into its original source code. In the current safety landscape, MI represents the vital transition from “black-box” observation—monitoring inputs and outputs—to “white-box” transparency. This shift is the prerequisite for “enumerative safety,” the ambitious objective of auditing every internal feature to proactively mitigate deceptive or catastrophic behaviors before they manifest in deployment.
The core mission of MI can be distilled into three strategic pillars:
From Behavior to Computation: Moving beyond explaining what a model does to understanding the internal mathematical operations that produce the output.
Decomposition of Activation Spaces: Resolving high-dimensional, polysemantic spaces into discrete, interpretable features.
Circuit Mapping: Identifying the “computational subgraphs”—the specific weights and non-linearities—that connect early-layer features to high-level algorithmic conclusions.
So What? Mechanistic Interpretability provides the technical foundation for Microscope AI. By reverse engineering models that may surpass human capabilities, we can learn the novel world-models they have discovered. More critically, MI is our primary defense against deceptive alignment; it allows us to identify whether a model is performing a task via a safe algorithm or a deceptive one that merely appears compliant during black-box testing.
This evolution began not in language, but in the structured, visually grounded domains of vision and multimodal models
2. Phase I: Foundational Vision and the “Features as Neurons” Era
The early successes of MI in image models (InceptionV1) and multimodal architectures (CLIP) established the first frameworks for “features” and “circuits.” These researchers demonstrated that internal model components often correspond to human-legible concepts.
.3. Phase II: The Mathematical Framework and the Discovery of Induction Heads (2021)
The 2021 publication of “A Mathematical Framework for Transformer Circuits” strategically reframed the Transformer not as a sequence of layers, but as a residual stream. This stream acts as a high-dimensional “shared memory” that attention heads and MLP layers read from and write to via linear operations.
This framework introduced a critical mnemonic for understanding attention: the QK (Query-Key) circuit determines “where” to look, while the OV (Output-Value) circuit determines “what” information to copy.
The Induction Head Circuit
The standout discovery of this era was the Induction Head, a two-step algorithmic circuit that allows models to perform “strict induction” (e.g., continuing the pattern [A][B] ... [A] with [B]).
Previous Token Head (The “Where”): An early-layer head attends to the previous token and writes a feature to the residual stream: “The token at the position before me was [A].”
Induction Head (The “What”): A later-layer head searches for source tokens that possess the “token before me was [A]” feature. When the QK circuit finds a match, the OV circuit copies the current token’s value ([B]) to predict it will follow the current instance of [A].
This discovery explained the “Induction Phase Transition.” Researchers observed a sudden, non-convex “bump” in the loss curve during training. This shift marks the algorithmic moment when a model moves from simple bigram statistics to the generalized capability of In-Context Learning.
4. Phase III: Navigating the Superposition Crisis (2022)
As research moved into Transformers, the “Features as Neurons” hypothesis collapsed into the Superposition Crisis. Models often represent N features in M dimensions where N>M, treating features as directions in space rather than individual neurons.
Strategic Taxonomy of Superposition
Bottleneck Superposition: Occurs in low-dimensional spaces like the residual stream. Strategically, this is used for the storage of information across layers.
Neuron Superposition: Occurs within MLP layers. This represents the model performing computation in a compressed, non-orthogonal state, making it far harder to interpret.
Softmax Linear Units (SoLU)
To mitigate the “smuggling” of polysemantic features, researchers introduced the SoLU activation function. This architectural intervention uses lateral inhibition and activation sparsity to incentivize monosemanticity. By applying a softmax across neurons before the LayerNorm, SoLU forces neurons to compete, making “hidden” features more accessible to researchers without sacrificing the performance of standard GELU activations.
5. Phase IV: “Interpretability in the Wild” – The IOI Milestone (Late 2022)
The study of Indirect Object Identification (IOI) (e.g., “John and Mary... John gave it to [Mary]”) was the first deep-dive into a complex, algorithmic task in a production-level model (GPT-2 Small).
The IOI Circuit Components
Duplicate Token Heads: Identify which names in the context have appeared previously.
S-Inhibition Heads: Use information from Duplicate Token Heads to inhibit the model’s focus on the repeated “Subject” (John).
Name Mover Heads: Attend to the remaining name (Mary) and “copy” it to the output.
The Strategic “So What?”: Redundancy and Safety IOI research identified Backup Name Mover Heads—redundant heads that only activate if the primary heads are ablated. This discovery is a direct challenge to “safety by ablation.” It implies that simply pruning a “bad” circuit is insufficient, as models have latent, compensatory paths that can “kick in” to restore behavior. Consequently, MI has moved toward Causal Scrubbing, a rigorous standard of evidence that operationalizes the localization of behavior by systematically testing every path in a computational subgraph.
6. Phase V: The “200 Concrete Open Problems” Taxonomy (Jan 18, 2023)
In early 2023, Neel Nanda published the “200 Concrete Open Problems” (COP) sequence, providing a roadmap to scale MI beyond “Streetlight Interpretability” (looking only where it is easy). This taxonomy categorizes model features into a biological analogy of sensory and motor functions.
Taxonomy of Model Features
De-tokenization (Sensory Neurons): Early layers that convert raw token inputs into internal formats.
Example: Disambiguation neurons for the token “ die”. A model must determine if it is the English verb, or the Dutch/German/Afrikaans article, using context to “sensory” process the raw input.
Example: Merging “Donald” and “Trump” into a single entity representation.
Conceptual Processing (Middle Layers):
Example: “Sophisticated” neurons that detect abstract categories, such as “numbers referring to groups of people.”
Re-tokenization (Motor Neurons): Late layers that convert conceptual conclusions back into token actions.
Example: After deciding the concept is “nappies,” motor neurons predict the specific multi-token sequence “n|app|ies”.
Memory Management: Neurons with high negative cosine similarity between input and output weights, effectively “deleting” information from the residual stream once it is no longer useful.
Critical Pitfalls
The COP sequence identified key standard-of-evidence traps:
The Interpretability Illusion: A neuron may appear monosemantic on one dataset (e.g., song lyrics) but fire for a completely different concept on another (e.g., historical dates).
Neuron Splitting: As models scale, a single high-level feature (e.g., “hexadecimal”) may split into multiple hyper-specific neurons (e.g., “the digit 3 in hexadecimal”).
As of early 2023, Mechanistic Interpretability is establishing its boundaries as a systematic science. While still in its “artisanal” stage, the transition toward automated circuit discovery and formal evidence standards like causal scrubbing marks the field’s maturity as a pillar of AI Alignment.
FAQ
1. Fundamental Philosophy and Scope
Mechanistic Interpretability (MI) is the engineering discipline of reverse-engineering neural networks from their learned weights and activations into human-understandable algorithms. While traditional AI research treats models as statistical “black boxes,” MI treats them as compiled code that can be decomposed into discrete logic. This is strategically vital to the “Microscope AI” vision: the ability to verify a model’s internal reasoning and safety properties without relying solely on its output. For the safety community, MI is the primary path toward enumerative safety, moving us from hope to a rigorous science of verification.
What distinguishes Mechanistic Interpretability from “Black-box Interpretability”? Standard “black-box” techniques (like saliency maps) focus on the relationship between inputs and outputs—effectively a form of behavioral psychology. They tell us that a model is looking at a specific pixel. MI is “white-box” interpretability. It investigates the internal circuitry to determine how the computation occurs. If black-box interpretability is psychology, MI is hardware reverse-engineering or digital forensics.
What is the strategic difference between an activation and a weight? Weights are the learned, static parameters that “program” the model’s behavior. Activations are the intermediate values computed during a specific forward pass. MI researchers analyze weights to find the underlying code and study activations to see how that code executes on specific data.
The “So What?” Layer: From Heuristics to Compiled Code
The transition from identifying that a model performs a task to understanding how it performs it is professionally imperative. Without mechanistic understanding, safety is a fuzzy heuristic. With it, we gain the ability to perform compiled code analysis on a model’s weights. This allows us to detect deceptive alignment—a scenario where a model behaves safely during training but hides “deceptive” circuits that only activate in unmonitored production environments. MI provides the technical basis for a safety audit that is robust against adversarial models.
Understanding these high-level goals is the “why.” To address the “how,” we must analyze the transformer as a physical medium of computation.
2. The Transformer as a Computational Medium
To reverse-engineer a Transformer, one must view the Residual Stream not just as a series of layers, but as a shared bandwidth or high-dimensional memory. Each layer reads from the stream, performs a small incremental update, and writes back. Viewing the model as a chain of independent layers is a mistake; it is more accurate to see it as a collection of components performing weight folding and incremental refinements to a shared state. In practice, researchers often “fold” LayerNorm parameters into the linear layers to simplify the linear algebra of the reverse-engineering process.
How do the different layers of the Transformer interact? The model is composed of specialized units that manage information routing and processing.
What are Tokenization, Embedding, and the Unembed?
Tokenization: Breaking language into sub-word units.
Embedding (WE): A lookup table converting tokens into vectors. In some models, the first MLP layer acts as an “extended embedding.”
Unembed (WU): The linear map projecting the final residual stream state back into the vocabulary space to predict the next token.
The “So What?” Layer: Features as Directions
A fundamental shift in MI is the “Features as Directions” hypothesis. We posit that concepts are not represented by single neurons, but as specific directions in activation space. Because the core of a Transformer is linear algebra, representing features as directions allows the model to extract and combine concepts via simple projections. This moves our focus from a “neuron-centric” view to a geometric one, where the activation space is a coordinate system of learned concepts.
This shared bandwidth allows for incredible performance, but it introduces the central hurdle of interpretation: the interference pattern known as Superposition.
3. Features, Neurons, and the Challenge of Superposition
There is a strategic tension between model performance and interpretability. To track more features than they have dimensions (e.g., 10,000 features in a 768-dimensional stream), models utilize Superposition—a form of lossy compression. This creates an overcomplete basis, where features are not orthogonal and “leak” into one another, making the internal logic difficult to untangle for a human auditor.
What is the difference between Monosemanticity and Polysemanticity?
Monosemanticity: A neuron (or direction) that corresponds to a single, clear feature.
Polysemanticity: A neuron that activates for multiple, unrelated features (e.g., a “dice and poets” neuron).
The Interpretability Illusion: A dangerous pitfall where a neuron appears monosemantic on one dataset but is actually multimodal. For example, a BERT neuron might seem to represent “historical dates” in trivia, but also activates for “objects in containers” in fiction.
How do GELU and SoLU activations impact interpretability? Most models use GELU, which permits significant superposition. SoLU (Softmax Linear Units) is designed to combat polysemanticity through Lateral Inhibition. By applying a softmax across the neurons before the LayerNorm, SoLU forces neurons to compete, encouraging the model to represent features in a more monosemantic, neuron-aligned basis.
The “So What?” Layer: The Quest for a Privileged Basis
The researcher’s workflow is complicated by Neuron Splitting (general features in small models becoming hyper-specific in large ones) and Equivariance (families of neurons representing symmetric concepts like different hexadecimal digits). Because of superposition, there is no “standard” basis that explains everything. We are searching for a privileged basis—a coordinate system where the axes align with human concepts. Understanding these geometric clusters is the only way to move beyond “fuzzy” intuition toward a rigorous map of a model’s world-model.
As we move beyond individual neurons, we observe how these units compose into high-level, algorithmic structures called Circuits.
4. High-Value Circuits: Induction and Indirect Object Identification (IOI)
Circuits are the computational sub-graphs—sets of heads and neurons—that perform specific tasks. The Induction Head is the landmark discovery of MI, proving that models develop universal, algorithmic solutions for in-context learning.
How does an Induction Circuit work via QK and OV circuits? Induction is a two-step process of K-composition across different layers:
Previous Token Head: Its OV-circuit writes a feature to the residual stream at the current position: “The token before me was A.”
Induction Head: Its QK-circuit searches for that specific “token before me was A” feature in the past. When it finds it at a position followed by token B, the Induction Head’s OV-circuit copies the identity of token B to the current position to predict it as the next token.
What is the Indirect Object Identification (IOI) circuit? The IOI circuit (e.g., “John and Mary... John gave the bag to [Mary]”) involves:
Name Mover Heads: These attend to the names in the text and copy them to the output logits.
S-Inhibition Heads: These detect which name is the “subject” (the repeated name) and actively inhibit Name Movers from attending to that name.
Negative Name Movers: Crucially, some heads actively suppress the correct answer. This “negative” logic shows the model is balancing multiple internal hypotheses, complicating safety audits.
The “So What?” Layer: The Backup Name Mover Phenomenon
Researchers have identified Backup Name Movers—redundant circuits that only activate if primary heads are ablated (removed). This discovery has massive strategic implications: a model might have “hidden” circuits that only reveal themselves when primary safety systems are tampered with. This redundancy is a major hurdle for safety audits; it suggests that “disabling” a harmful behavior by removing a single circuit may be insufficient, as the model may simply switch to a backup.
To prove the existence of these circuits, we cannot rely on simple observation; we must use causal interventions.
5. Methodologies for Empirical Investigation
In MI, simply looking at “max activating dataset examples” is insufficient—it often leads to the Interpretability Illusion. To build a rigorous science, we must intervene in the computation to prove causality.
What are the primary techniques for causal investigation?
Activation Patching: We take a “clean” run and a “corrupted” run (where input is changed). We then “patch” activations from the clean run into the corrupted run. if the output switches back to the clean answer, we have causally localized the computation.
Direct Logit Attribution (DLA): We project a specific component’s output onto the unembedding (WU). This measures that component’s direct, immediate contribution to the final logit score.
Logit Lens: Unlike DLA, the Logit Lens involves “reading” the state of the entire residual stream at an intermediate layer by passing it through the final unembedding prematurely.
The “So What?” Layer: Falsifying Hypotheses with Neuroscope
Tools like Neuroscope and Interactive Neuroscope allow researchers to transition from passive observation to active adversarial testing. By injecting noise, replacing tokens with synonyms, or deleting words, we can attempt to falsify hypotheses. If we think a neuron is a “Base64 neuron,” we must test it against hexadecimal and random noise. This rigorous falsification is the only professional standard for ensuring our interpretations aren’t just “just-so stories.”
Despite these tools, MI remains a young field with high research debt and critical open problems.
6. The Roadmap: Open Problems and Scaling
MI faces significant “research debt.” The “200 Concrete Open Problems” is a strategic call to action to move MI from a hobbyist pursuit to a robust safety science.
What are the most critical categories of Open Problems?
Conceptual Neurons: Identifying “middle-layer” neurons that represent abstract concepts rather than simple sensory inputs.
Memory Management vs. Signal Boosting: We must distinguish between neurons that “delete” information (identified by high negative cosine similarity between Win and Wout) and those that “amplify” important features (identified by high positive cosine similarity).
Context Neurons: Identifying neurons that track the “mode” of text, such as Base64 strings, Hexadecimal code, Python-specific markers, or Section Titles in LaTeX.
The “So What?” Layer: Moving Beyond “Streetlight Interpretability”
A common critique of MI is “Streetlight Interpretability”—focusing only on easy, “bright” problems like Induction Heads. To be strategically relevant for AI Safety, MI must scale. This requires automated interpretability tools—AI systems that can reverse-engineer other AI systems. We cannot hand-label every circuit in a trillion-parameter model. Developing the automated capacity to map these circuits is the professional imperative for the alignment community to ensure a safe, transparent, and verifiable AI future.
Table of Contents with Timestamps
Reverse-Engineering the AI Mind
00:00 — Cold Open — The Heliox tagline and mission statement open the episode.
00:24 — The Research Stack — Introducing five years of mechanistic interpretability literature, from 2021 foundational work to 2026 preprints.
01:01 — The Black Box Paradox — We rely on neural networks daily, yet even their creators can’t see how they reason internally.
02:00 — Meet Neil Nanda — The episode’s throughline: one researcher’s career as a map of the whole field.
03:10 — The Blueprint Objection — If humans wrote the code, why do we need to reverse-engineer it?
05:23 — Cambridge to Anthropic — Nanda’s mathematical training and mentorship under Chris Olah.
06:47 — The Decompilation Dream — Early interpretability’s founding hypothesis: models as compiled binaries waiting to be decoded.
08:00 — Transformer Circuits — The landmark 2021 paper that gave researchers a real vocabulary for the transformer’s internals.
08:42 — Induction Heads — How models learn to bind and recall new information within a single conversation.
11:25 — Building a Movement — Neuroscope, 200 Open Problems, and the effort to bootstrap a research community.
12:34 — Universality — The hypothesis that all sufficiently trained networks converge on the same circuits, echoing convergent evolution.
14:05 — Features and Circuits — The nouns and verbs of a model’s internal vocabulary.
15:59 — The IOI Circuit — Mapping, by hand, how a model tracks who did what to whom in a sentence.
17:05 — Detokenization and Retokenization — The model’s sensory and motor neurons, fusing and unpacking tokens into concepts.
18:26 — Streetlight Interpretability — A pointed critique: was the field only solving the easy, well-lit problems?
19:44 — Digital Phrenology and the Interpretability Illusion — Why a single neuron rarely means just one thing.
21:38 — Superposition — The 2022 discovery that models pack far more concepts than they have dimensions to hold them.
24:01 — The Suitcase Analogy — Sparsity, antipodal pairs, and why overlapping representations mostly work.
27:52 — Nanda’s Reckoning — The admission that the field’s most ambitious vision is, in his words, probably dead.
29:37 — Sparse Autoencoders — The technical fix that forces a model’s tangled representations back into legibility.
31:57 — GemmaScope and Attribution Graphs — Automating circuit discovery at scale, and the shift from understanding to applied safety.
34:43 — MAIA: The First Interpretability Agent — MIT’s 2024 system that runs its own experiments on other models.
36:03 — The Circuit Oracle — A multi-agent swarm — Scout, Dispatch, Merge, Build — built to traverse causal graphs too large for humans.
38:10 — Who Watches the Watchman — The confirmation-bias problem in AI systems interpreting other AI systems.
40:01 — The Memory Bottleneck — Why retrieval-based memory breaks down at the scale of a full attribution graph.
41:32 — The LLM Wiki — Andrej Karpathy’s 2026 proposal to treat knowledge as compiled code, not retrieved fragments.
43:03 — Dreaming — Anthropic’s asynchronous framework for letting agents consolidate memory offline.
44:37 — The Immutability Safeguard — Why the raw experimental record is never allowed to be altered.
46:44 — Closing the Arc — From manual circuit-mapping to dreaming machines, in five years.
47:57 — Final Thought — If universality holds for AI, what does that imply about the human mind?
Index with Timestamps
4. Index
Agentic interpretability, 34:43
Anthropic, 06:06
Antipodal pairs, 24:06
Asynchronous epistemology, 44:11
Attribution graph, 32:00
Chris Olah, 06:11
Circuit Oracle, 36:03
Confirmation bias, 38:46
Convergent evolution, 13:03
Curse of dimensionality, 21:38
Decompilation analogy, 07:08
Detokenization neurons, 17:06
Dreaming framework, 42:48
Features (definition), 14:14
GemmaScope, 31:34
Indirect object identification (IOI) circuit, 16:09
Induction heads, 08:52
Interpretability illusion, 20:21
Karpathy, Andrej, 41:34
LLM Wiki, 41:39
MAIA (Multimodal Automated Interpretability Agent), 34:56
Mechanistic interpretability (defined), 02:07
Neel Nanda, 02:24
Neuroscope, 12:08
Phrenology analogy, 19:44
Privileged basis problem, 26:44
Retrieval-augmented generation (RAG), 40:38
Sleeper agents, 33:30
Sparse autoencoders (SAEs), 29:56
Sparsity, 24:46
Streetlight interpretability, 18:36
Superposition, 22:04
Toy Models of Superposition (2022 paper), 21:48
Transformer Circuits paper (2021), 08:05
Universality hypothesis, 12:33
Poll
Post-Episode Fact Check
Neel Nanda’s background — Confirmed. Nanda earned a pure mathematics degree from Cambridge, graduating in 2020, then worked at Anthropic on interpretability research under Chris Olah (2021–2022) before moving to Google DeepMind, where he now leads the mechanistic interpretability team. Public sources place his transition to DeepMind around 2022–2023, not precisely pinned to “age 26” as stated in the episode, though the timeline is broadly accurate — he was in his mid-twenties.
“A Mathematical Framework for Transformer Circuits” (2021) — Confirmed. Neel Nanda is a listed co-author alongside Nelson Elhage, Catherine Olsson, Chris Olah, and the broader Anthropic team. This paper did introduce the vocabulary and discovery of induction heads described in the episode.
Nanda’s public admission that the field’s ambitious vision is “probably dead” — Confirmed. Nanda made this statement explicitly in a 2025 interview (80,000 Hours podcast), describing his own trajectory as moving “from idealistic ambition to optimistic pragmatism.”
“Toy Models of Superposition” (Anthropic, 2022) — Confirmed as a real, foundational paper formalizing superposition and polysemanticity in neural networks.
MAIA (Multimodal Automated Interpretability Agent) — Confirmed. Published by MIT CSAIL researchers (Tamar Rott Shaham, Sarah Schwettmann, and colleagues) in April 2024, presented at ICML 2024. The episode’s description of its capabilities — hypothesis generation, experiment design, iterative refinement — matches the published research.
Circuit Oracle — Confirmed as a real, recent (June 2026) research paper describing a multi-agent system for automating attribution graph analysis, evaluated on tasks including detecting spurious features and eliciting hidden knowledge. The specific internal tool names “Scout,” “Dispatch,” “Merge,” and “Build” as described in the episode are a dramatized simplification for audio storytelling rather than verified terminology from the paper itself, and listeners should treat those specific labels as illustrative rather than as direct quotes from the source.
GemmaScope — Confirmed as a real, large-scale open release of sparse autoencoders across the Gemma 2 model family, led by DeepMind’s interpretability team.
Andrej Karpathy’s “LLM Wiki” concept — Confirmed. Karpathy published this idea via an X post and GitHub gist in April 2026, describing a pattern of compiling agent knowledge into a persistent, cross-referenced markdown wiki rather than relying solely on retrieval-augmented generation.
Anthropic’s “Dreaming” framework — Confirmed. Anthropic introduced “Dreaming” as a research-preview feature for Claude Managed Agents on May 6, 2026, at its Code with Claude developer conference. The feature does operate as a scheduled, asynchronous background process that reviews past session transcripts (up to 100 sessions, matching the episode’s description) to merge, update, and surface patterns in an agent’s memory store.
Overall assessment: The episode’s core historical and technical narrative is well-supported by publicly verifiable sources. The primary area of caution is the specific internal architecture described for the Circuit Oracle’s four named sub-tools, which appears to be a narrative simplification of the underlying research rather than a literal citation.
Image (3000 x 3000 pixels)
Mind Map
Comic
Street Art
Infographics
