Mechanistic interpretability
Reverse-engineering neural networks
- 2 min read
- 3 chapters
Related on Lably
Through Explainable artificial intelligence, Neural network (machine learning) and Large language model
History
The term "mechanistic interpretability" was coined by Chris Olah, co-founder of Anthropic, as a description of his work in circuit analysis as opposed to usual methods in interpretable AI. Circuit analysis attempted to completely characterize individual features and circuits within models, while the broader field tended towards gradient-based approaches like saliency maps.
Before circuit analysis, work in the subfield combined various techniques such as feature visualization, dimensionality reduction, and attribution with human-computer interaction methods to analyze models like the vision model Inception v1.
Sign in to keep reading
2 more chapters are free with a Lably account.
Key concepts
Mechanistic interpretability aims to identify structures, circuits or algorithms encoded in the weights of machine learning models. This contrasts with earlier interpretability methods that focused primarily on "black box" explanations.
Linear representation hypothesis
This hypothesis suggests that high-level concepts are represented as linear directions in the activation space of neural networks. Empirical evidence from word embeddings and large language models supports this view, although it does not hold up universally.
Methods
Mechanistic interpretability employs causal methods to understand how internal model components influence outputs, often using formal tools from causality theory.
In AI safety, mechanistic interpretability is used to understand and verify the behavior of complex AI systems, and to attempt to identify potential risks such as AI misalignment.
Sparse autoencoders
A sparse autoencoder (SAE) is a model trained to disentangle neural network activations into sparse representations. The learned dimensions often represent simple, human-understandable concepts. The technique was applied to large language model interpretability by Anthropic.
Features and circuits
A circuit in a neural network is composed of causal chains of feature activations. By mapping out what circuits lead to what downstream consequences, as well as by activating and inhibiting circuits, one can analyze how a neural network (such as an LLM) reaches a given result from a given input.