Mechanistic interpretability · Computational neuroscience
Richard Jeevash.
I like an explanation I can try to break. And a quiet scene that somehow breaks me.
I test whether explanations of neural networks hold up when you intervene on them, in language models and in models of the fly brain.
Findings go up before peer review, clearly labelled. Here’s why.
The question running through it
Does the explanation survive the intervention?
Language models
What an activation patch really moves, and when a test can tell a right explanation from a wrong one.
Models of the fly brain
Wiring-based models of the fly’s brain: does silencing a cell type in the model match silencing it in the fly?
Latest findings
All researchCuriosity, with a paper trail.
How small models indent code, write ordinals and keep capitals
How small models indent Python, write “112th” and keep writing in capitals, mostly with separate parts.
How a small model breaks lines: a partial account
A small model counts a line’s characters by how thin its attention to the line start gets.
Notes
All notesMostly coffee. Some questions.
Why I post findings before peer review
Interpretability moves fast and much of it isn’t settled. Posting checked findings early, clearly labelled, seems more useful than holding them back for a year.
How I run a study
Freeze the plan, save every output, generate every number from what was saved, and report what didn’t work. The routine behind each paper on this site.
Tools
- Open source
Logogram
A local-first workbench for causal experiments inside language models. Change one activation, watch what changes.
- Open source
DigitalFly
A virtual fruit fly you can poke: a simulated body steered by a brain you can inspect, silence and tune.