Working paperNot peer-reviewed12 November 2026
Small decisions, mostly separate parts: how small language models indent code, choose ordinal suffixes and write in capitals
In short Three small decisions, mostly separate mechanisms: a few late heads copy the indent of the block being closed, an early head and a late MLP apply the “teen, so th” rule, and capitals are added late along one direction. Two late heads have a small second role; the groups that do most of each task’s work are separate.
- 0.893of dedents past the innermost block where the chief block-end head looked hardest at the right header
- 0.981of positions where adding one direction turned the next word to capitals (Gemma-4-E2B; 0.956 in Qwen3-1.7B)
- 2 late headswith a small second role: L17H6 in line-breaking and indentation, L11H9 in the ordinal suffix and the dedent after a terminal statement
The question
Much of what a language model writes is decided by small rules of form: how far to indent the next line of Python, whether a number takes st or th, whether the next word carries on a passage written in capitals. Small models get all three mostly right. This paper asks how, in small open-weight base models, and then asks a question only a set of studies can answer: do the mechanisms share parts?
1. Indentation: copy the indent of the block you’re closing
An unmarked dedent in Qwen3-0.6B. The header the chief block-end head attends to sets the next line’s indent.
- Two routes. After return, raise, break, continue or pass, the keyword is read directly from the end of the line, through many heads. Other dedents depend on a small set of late heads, chief among them L23H2.
- The header it looks at is the choice. Where the block to return to was not the innermost one, the open header L23H2 attended to most was the right one in 0.893 of cases.
- Steering it moves the prediction. Pointing its attention at another open header raised that header’s indent by +2.72 nats in natural code; another head of the same layer moved it by +0.08.
- A second model. OLMo-2-1B has both routes too, with head L13H1 as its block-end reader.
2. Ordinal suffixes: an early copy and a late override
To write “112th” in Qwen3-0.6B, two writers that never consult each other meet in the output, and the teen override wins.
- The rule isn’t computed as an “if”. An early head, L0H2, brings the tens digit to the last digit and MLP0 reads it. Late in the model, MLP24 writes “teen, so th” without knowing which suffix it overrides, while MLP18 separately writes the last digit’s own suffix.
- Tokens decide where it breaks. Where OLMo-2’s tokeniser puts the tens digit in another token, the signal arrives too faint, and a thousands separator repairs the errors. GPT-2 and Pythia-1B fail the same way on the few numbers their tokenisers split, though on only ten distinct teen numbers each: too few to show the failure holds beyond them.
OLMo-2-1B falls only at four and seven digits, where its chunks of three leave the last digit in a token of its own. The thousands separator removes the dip. Qwen3-0.6B, which gives every digit its own token, shows no such pattern.
Teen AUROC. OLMo-2 lines are held-out results; Qwen3-0.6B is from exploration. From the paper’s Figure 2.
The numbers
| OLMo-2-1B | OLMo-2-1B with a separator | Qwen3-0.6B | |
|---|---|---|---|
| Digits: 2 | – | – | 1.000 |
| Digits: 3 | 1.000 | – | 0.968 |
| Digits: 4 | 0.775 | 0.989 | 0.921 |
| Digits: 5 | 0.999 | 0.974 | 0.915 |
| Digits: 6 | 0.975 | 0.918 | 0.896 |
| Digits: 7 | 0.729 | 0.953 | 0.903 |
3. Capitals: lower case first, capitals last
Earlier work found that an in-context “capitalise this word” function is applied late, along one vector (Merullo et al., 2024). This study looks at a whole passage written in capitals: where its case is carried, and which parts write and use it.
Writing a passage in capitals, in Qwen3-1.7B. The direction sits at layer 22 here, and at layer 24 in Gemma-4-E2B.
On held-out passages in capitals, the logit lens reads the coming word in lower case at the direction layer (0.753 of positions). The capitals form takes over for good only in the last three layers. Gemma-4-E2B shows the same shape.
Descriptive. Read across depth, the lens leans towards lower-case tokens in the first layers and shows the coming word only from mid depth: “lower case first” means that once the word can be read, it reads in lower case. From the paper’s Figure 4.
The numbers
| The word in lower case beats its capitals | Another word in lower case beats them | |
|---|---|---|
| Layer 0 | 0.791 | 0.750 |
| Layer 1 | 0.772 | 0.700 |
| Layer 2 | 0.766 | 0.707 |
| Layer 3 | 0.683 | 0.687 |
| Layer 4 | 0.708 | 0.695 |
| Layer 5 | 0.736 | 0.703 |
| Layer 6 | 0.753 | 0.699 |
| Layer 7 | 0.768 | 0.704 |
| Layer 8 | 0.771 | 0.707 |
| Layer 9 | 0.783 | 0.699 |
| Layer 10 | 0.804 | 0.737 |
| Layer 11 | 0.818 | 0.724 |
| Layer 12 | 0.811 | 0.722 |
| Layer 13 | 0.813 | 0.715 |
| Layer 14 | 0.811 | 0.696 |
| Layer 15 | 0.826 | 0.706 |
| Layer 16 | 0.827 | 0.611 |
| Layer 17 | 0.842 | 0.544 |
| Layer 18 | 0.836 | 0.494 |
| Layer 19 | 0.647 | 0.152 |
| Layer 20 | 0.729 | 0.165 |
| Layer 21 | 0.709 | 0.118 |
| Layer 22 | 0.753 | 0.127 |
| Layer 23 | 0.745 | 0.120 |
| Layer 24 | 0.507 | 0.054 |
| Layer 25 | 0.211 | 0.019 |
| Layer 26 | 0.068 | 0.010 |
| Layer 27 | 0.002 | 0.001 |
- One direction switches the case. Adding it to a lower-case passage made the next word upper case at 0.981 of positions in Gemma-4-E2B and 0.956 in Qwen3-1.7B. Random directions of the same size flipped at most 0.003.
- A few parts write and carry it, and the switch needs the MLPs after it. In Qwen3-1.7B, MLPs 19–22 and head L19H0 write it, and four heads carry the passage’s case. Freezing the MLPs after the direction layer stopped the flip: they are needed for it, which does not show they do nothing else.
Acronyms and words with an initial capital do not use it: removing the direction leaves them in capitals. Headings written in capitals do: removing it kept all capitals inside them at only 0.669 and 0.768.
From the paper’s Tables 3 and 23 (A1–A3).
The numbers
| Qwen3-1.7B | Gemma-4-E2B | |
|---|---|---|
| Acronyms | 1.000 | 1.000 |
| Initial capitals | 0.968 | 0.982 |
| Headings in capitals | 0.669 | 0.768 |
Do the three share parts?
The cross-study test in Qwen3-0.6B. Capitals were tested in Qwen3-1.7B, with heads matched by index; the line-breaking parts come from the companion paper on line-breaking.
A separate frozen test asked whether the groups each study named are really the same parts. It found less sharing than earlier reports of component reuse:
- The groups that do most of each task’s work are separate. Switching them off harms the other tasks no more than random groups of the same layers do.
- Two late heads have a small second role. Without L17H6, the model kept only 0.706 of the effect of line width on where it breaks a line, against 0.988 to 1.036 for random heads of its layer, and the head adds a little to the dedent after return. L11H9 helps choose the ordinal suffix and adds to the dedent after a terminal statement: a role first noticed after the cross-study test, then confirmed by a frozen test on new material (−0.027, below all twenty random heads of its layer).
- General early movers. The early heads the studies named, L0H2 and L0H3, move the previous or the current token in every task: general parts, not task parts.
- What “separate” means here: within bounds fixed in advance, on these measures, at the decision positions. Comparing every value with twenty layer-matched random groups is post hoc, except for L11H9’s second role, which the third round tested prospectively. Sharing through MLPs and the residual stream is not ruled out.
How it was tested
- 01
Gate
A model was studied only if it did the task well. Pythia-410M and GPT-2 failed the indentation gate; Qwen3-0.6B failed the capitals gate.
- 02
Explore
Under a frozen charter: ablation, patching, attention, the logit lens and steering, until a candidate mechanism appeared.
- 03
Freeze
Predictions, thresholds, controls and code written down and frozen. Every named set is compared with twenty random sets from the same layers.
- 04
Test once
On material sealed before any model read it, two rounds per study. A third round re-ran every code-based test on code that can be shared.
The predictions were written after exploration, so the counts below record that each account repeated on held-out material, not that it foresaw effects nobody had looked for. A test is one prediction read in one model (and, for ordinals, at one number length).
- 9 of 9indentation, round 1
- 25 of 25indentation, round 2
- 34 of 34indentation, round 3, on shareable code
- 14 of 14ordinals, round 1
- 39 of 41ordinals, round 2
- 19 of 19capitals, round 1
- 23 of 23capitals, round 2
- 22 of 22shared parts
- 15 of 15shared parts, round 3
The two failed tests are one ordinal prediction at its two number lengths, in Qwen3-1.7B: that its early previous-token heads would leave the last digit’s pull intact. They removed much of it too (0.548 and 0.233 kept, against the 0.55 required). In that model the two kinds of early head divide the work less cleanly.
What this doesn’t show
- Small base models (the largest, Gemma-4-E2B, has 2.3 billion effective parameters), English text, Python only for indentation, numbers up to ten digits, and Wikipedia paragraphs and headings for capitals. The ordinal path was traced only in Qwen3-0.6B.
- The accounts are partial. What tells the block-end head where to look is spread over many components, and the ordinal signal between MLP0 and MLP24 is spread thinly and untraced. Removing the capitals direction left about half of the words in capitals, so case is also carried in other ways.
- Mean ablation, direct attribution, the logit lens and patching choices all have known weaknesses. Random-set controls bound these concerns but do not remove them.
- The cross-study test used one model family, with capitals heads matched by index, and ablated at the decision positions only. The third round re-ran only the code-based tests.
- Held-out material is held out from these analyses, not necessarily from the models’ training data. The code and the Wikipedia articles come from repositories and pages created after the models were trained, but these can contain older code and text.
How the study went
For each study: a frozen charter, exploration, then two frozen rounds on sealed held-out material, the second tracing the first round’s account into its parts. Then a frozen cross-study test of shared parts.
A measurement error that dropped points after blank lines was found and fixed by a dated amendment; it changed no gate verdict. A post hoc weight comparison showed that OLMo-2’s published checkpoints are not ancestors of the released model, and the inference that depended on it was withdrawn.
A third indentation round, frozen before any model read its material, re-ran every code-based test on permissively licensed and newly generated code. All held again, and it confirmed L11H9’s second role, first noticed after the cross-study test.
This version, with the code and data archive: 34 pages.
Read it, cite it
- Status
- Working paper, not peer-reviewed
- This version
- 12 November 2026
- Field
- Language models
- Models
- Qwen3-0.6B, Qwen3-1.7B, OLMo-2-1B, Gemma-4-E2B, and others
- Archive
- doi:10.5281/zenodo.24150049
Read the abstract
Small language models make many small formatting and morphology decisions correctly: how far to indent the next line of Python, whether a number takes st or th, whether the next word continues a passage written in capitals. We asked how they make three of these decisions, and then whether the mechanisms share parts. Each study explored under a frozen charter, then wrote down what the explored account predicts and tested those predictions, with thresholds and layer-matched random controls, once on held-out material, so the tests show whether each account repeats on new material. Indentation (Qwen3-0.6B-Base and OLMo-2-0425-1B): dedents reach the model by two routes. After return and similar statements the keyword is read directly from the decision token through many heads. Unmarked dedents depend on a small set of late heads, chief among them one that attends to the line-ending token of an open block’s header and copies that header’s indent; which open header it attends to sets how far to dedent, and steering its attention moves the prediction. Ordinal suffixes (Qwen3-0.6B-Base, OLMo-2-0425-1B and five further models): to write “112th”, Qwen3-0.6B-Base uses an early previous-token head to bring the tens digit to the last digit; the first MLP reads it, and a late MLP writes a teen override towards th that does not consult the last digit, while a different MLP writes the last digit’s own suffix. In OLMo-2-0425-1B, where the tokeniser can put the tens digit in another token, the signal arrives too faint, and a thousands separator repairs the errors; early previous-token heads carry part of the check in three further models. In the ordinal study’s second round two tests failed, the two number lengths of one prediction. Capitals (Qwen3-1.7B-Base and Gemma-4-E2B): the next word is chosen in a lower-case-like form and turned into capitals in the last layers along one direction, which is written by a few late components and fed by heads that carry the passage’s case; the switch to capitals needs the MLPs after it. A prospective cross-study test (line-breaking, indentation and ordinals in Qwen3-0.6B-Base; capitals in Qwen3-1.7B-Base, with heads matched by index) finds less sharing than earlier reports of component reuse. The early heads the studies named are general movers of the previous or the current token. Of the later heads, two have a small second role: L17H6 is shared between line-breaking and indentation, and L11H9 serves both the ordinal suffix choice and the dedent after a terminal statement, a role first noticed after the cross-study test and then confirmed by a frozen test on new material. The groups that do most of each task’s work harm the other tasks no more than random groups do. A third round re-ran every code-based test of the indentation and cross-study work on permissively licensed and newly generated code, which can be redistributed, and all of them held again.
BibTeX
@misc{jeevash2026small,
title = {{Small decisions, mostly separate parts: how small language models indent code, choose ordinal suffixes and write in capitals}},
author = {Jeevash, Richard},
year = {2026},
note = {Working paper, not peer-reviewed. Code and data: doi:10.5281/zenodo.24150049},
url = {https://richardjeevash.com/research-small-decisions.html}
}