Richard Jeevash

Working paperNot peer-reviewed12 November 2026

Small decisions, mostly separate parts: how small language models indent code, choose ordinal suffixes and write in capitals

In short Three small decisions, mostly separate mechanisms: a few late heads copy the indent of the block being closed, an early head and a late MLP apply the “teen, so th” rule, and capitals are added late along one direction. Two late heads have a small second role; the groups that do most of each task’s work are separate.

Read the paper (PDF, 34 pages)CodeCode and data archiveCite

The question

Much of what a language model writes is decided by small rules of form: how far to indent the next line of Python, whether a number takes st or th, whether the next word carries on a passage written in capitals. Small models get all three mostly right. This paper asks how, in small open-weight base models, and then asks a question only a set of studies can answer: do the mechanisms share parts?

1. Indentation: copy the indent of the block you’re closing

def total(items): for x in items: if x > 0: s += x ⏎ return s chief block-end head · L23H2 From the end of the line, it attends to the line end of one open header and copies that header’s indent. Which open header it picks is how far the next line steps back. open headers are boxed; the next line starts under the one the head picked def total(items): for x in items: if x > 0: s += x ⏎ return s chief block-end head · L23H2 From the end of the line, it attends to the line end of one open header and copies that header’s indent. Which open header it picks is how far the next line steps back. open headers are boxed; the next line starts under the one the head picked

An unmarked dedent in Qwen3-0.6B. The header the chief block-end head attends to sets the next line’s indent.

  • Two routes. After return, raise, break, continue or pass, the keyword is read directly from the end of the line, through many heads. Other dedents depend on a small set of late heads, chief among them L23H2.
  • The header it looks at is the choice. Where the block to return to was not the innermost one, the open header L23H2 attended to most was the right one in 0.893 of cases.
  • Steering it moves the prediction. Pointing its attention at another open header raised that header’s indent by +2.72 nats in natural code; another head of the same layer moved it by +0.08.
  • A second model. OLMo-2-1B has both routes too, with head L13H1 as its block-end reader.

2. Ordinal suffixes: an early copy and a late override

one token per digit · Qwen3-0.6B 1 1 2 L0H2 copies it copy of the tens digit → MLP0 → … → MLP24: teen, so th last digit → L0H3 → MLP18: 2, so nd 112th they meet only here; the override wins chunks of three digits · OLMo-2-1B 111 2 the tens digit sits in another token: the signal arrives too faint, and teens go wrong 1 , 112 a thousands separator puts both digits in one token, and the errors go away one token per digit · Qwen3-0.6B 1 1 2 L0H2 copies it copy of the tens digit → MLP0 → … MLP24: teen, so th last digit → L0H3 → MLP18: 2, so nd 112th they meet only here; the override wins chunks of three digits · OLMo-2-1B 111 2 the tens digit sits in another token: the signal arrives too faint, and teens go wrong 1 , 112 a thousands separator puts both digits in one token, and the errors go away

To write “112th” in Qwen3-0.6B, two writers that never consult each other meet in the output, and the teen override wins.

  • The rule isn’t computed as an “if”. An early head, L0H2, brings the tens digit to the last digit and MLP0 reads it. Late in the model, MLP24 writes “teen, so th” without knowing which suffix it overrides, while MLP18 separately writes the last digit’s own suffix.
  • Tokens decide where it breaks. Where OLMo-2’s tokeniser puts the tens digit in another token, the signal arrives too faint, and a thousands separator repairs the errors. GPT-2 and Pythia-1B fail the same way on the few numbers their tokenisers split, though on only ten distinct teen numbers each: too few to show the failure holds beyond them.
Fig. 01 Teen errors follow token boundaries: how well each model tells teens apart, by number length
0.60.70.80.91.0234567Digits in the numberTeen AUROCOLMo-2-1BOLMo-2-1B with a separatorQwen3-0.6BOLMo-2-1B, 3: 1.000OLMo-2-1B, 4: 0.775OLMo-2-1B, 5: 0.999OLMo-2-1B, 6: 0.975OLMo-2-1B, 7: 0.729OLMo-2-1B with a separator, 4: 0.989OLMo-2-1B with a separator, 5: 0.974OLMo-2-1B with a separator, 6: 0.918OLMo-2-1B with a separator, 7: 0.953Qwen3-0.6B, 2: 1.000Qwen3-0.6B, 3: 0.968Qwen3-0.6B, 4: 0.921Qwen3-0.6B, 5: 0.915Qwen3-0.6B, 6: 0.896Qwen3-0.6B, 7: 0.9030.60.70.80.91.0234567Digits in the numberTeen AUROCOLMo-2-1BOLMo-2-1B with a separatorQwen3-0.6BOLMo-2-1B, 3: 1.000OLMo-2-1B, 4: 0.775OLMo-2-1B, 5: 0.999OLMo-2-1B, 6: 0.975OLMo-2-1B, 7: 0.729OLMo-2-1B with a separator, 4: 0.989OLMo-2-1B with a separator, 5: 0.974OLMo-2-1B with a separator, 6: 0.918OLMo-2-1B with a separator, 7: 0.953Qwen3-0.6B, 2: 1.000Qwen3-0.6B, 3: 0.968Qwen3-0.6B, 4: 0.921Qwen3-0.6B, 5: 0.915Qwen3-0.6B, 6: 0.896Qwen3-0.6B, 7: 0.903

OLMo-2-1B falls only at four and seven digits, where its chunks of three leave the last digit in a token of its own. The thousands separator removes the dip. Qwen3-0.6B, which gives every digit its own token, shows no such pattern.

Teen AUROC. OLMo-2 lines are held-out results; Qwen3-0.6B is from exploration. From the paper’s Figure 2.

The numbers
OLMo-2-1BOLMo-2-1B with a separatorQwen3-0.6B
Digits: 2––1.000
Digits: 31.000–0.968
Digits: 40.7750.9890.921
Digits: 50.9990.9740.915
Digits: 60.9750.9180.896
Digits: 70.7290.9530.903

3. Capitals: lower case first, capitals last

Earlier work found that an in-context “capitalise this word” function is applied late, along one vector (Merullo et al., 2024). This study looks at a whole passage written in capitals: where its case is carried, and which parts write and use it.

a passage in capitals: “THE BRIDGE THAT CROSSES THE …” 0510152027 layer river the coming word, readable from mid depth, in lower case RIVER ℓ = 22 carriers L17H6, L18H0, L18H1, L19H0 bring the passage’s case the switch needs the MLPs after it MLPs 19–22 and L19H0 write one capitals direction by layer 22 a passage in capitals: “THE BRIDGE THAT CROSSES THE …” layer 0510152027 the coming word, readable from mid depth, in lower case river carriers L17H6, L18H0, L18H1, L19H0 bring the passage’s case MLPs 19–22 and L19H0 write one capitals direction by layer 22 ℓ = 22 the switch needs the MLPs after it RIVER capitals in the last three layers

Writing a passage in capitals, in Qwen3-1.7B. The direction sits at layer 22 here, and at layer 24 in Gemma-4-E2B.

Fig. 02 Lower case first, capitals last: the coming word through the layers of Qwen3-1.7B
0.0000.2500.5000.7501.0000510152027LayerShare of positionsdirection layerThe word in lower case beats its capitalsAnother word in lower case beats themThe word in lower case beats its capitals, 0: 0.791The word in lower case beats its capitals, 1: 0.772The word in lower case beats its capitals, 2: 0.766The word in lower case beats its capitals, 3: 0.683The word in lower case beats its capitals, 4: 0.708The word in lower case beats its capitals, 5: 0.736The word in lower case beats its capitals, 6: 0.753The word in lower case beats its capitals, 7: 0.768The word in lower case beats its capitals, 8: 0.771The word in lower case beats its capitals, 9: 0.783The word in lower case beats its capitals, 10: 0.804The word in lower case beats its capitals, 11: 0.818The word in lower case beats its capitals, 12: 0.811The word in lower case beats its capitals, 13: 0.813The word in lower case beats its capitals, 14: 0.811The word in lower case beats its capitals, 15: 0.826The word in lower case beats its capitals, 16: 0.827The word in lower case beats its capitals, 17: 0.842The word in lower case beats its capitals, 18: 0.836The word in lower case beats its capitals, 19: 0.647The word in lower case beats its capitals, 20: 0.729The word in lower case beats its capitals, 21: 0.709The word in lower case beats its capitals, 22: 0.753The word in lower case beats its capitals, 23: 0.745The word in lower case beats its capitals, 24: 0.507The word in lower case beats its capitals, 25: 0.211The word in lower case beats its capitals, 26: 0.068The word in lower case beats its capitals, 27: 0.002Another word in lower case beats them, 0: 0.750Another word in lower case beats them, 1: 0.700Another word in lower case beats them, 2: 0.707Another word in lower case beats them, 3: 0.687Another word in lower case beats them, 4: 0.695Another word in lower case beats them, 5: 0.703Another word in lower case beats them, 6: 0.699Another word in lower case beats them, 7: 0.704Another word in lower case beats them, 8: 0.707Another word in lower case beats them, 9: 0.699Another word in lower case beats them, 10: 0.737Another word in lower case beats them, 11: 0.724Another word in lower case beats them, 12: 0.722Another word in lower case beats them, 13: 0.715Another word in lower case beats them, 14: 0.696Another word in lower case beats them, 15: 0.706Another word in lower case beats them, 16: 0.611Another word in lower case beats them, 17: 0.544Another word in lower case beats them, 18: 0.494Another word in lower case beats them, 19: 0.152Another word in lower case beats them, 20: 0.165Another word in lower case beats them, 21: 0.118Another word in lower case beats them, 22: 0.127Another word in lower case beats them, 23: 0.120Another word in lower case beats them, 24: 0.054Another word in lower case beats them, 25: 0.019Another word in lower case beats them, 26: 0.010Another word in lower case beats them, 27: 0.0010.0000.2500.5000.7501.0000510152027LayerShare of positionsdirection layerThe word in lower case beats its capitalsAnother word in lower case beats themThe word in lower case beats its capitals, 0: 0.791The word in lower case beats its capitals, 1: 0.772The word in lower case beats its capitals, 2: 0.766The word in lower case beats its capitals, 3: 0.683The word in lower case beats its capitals, 4: 0.708The word in lower case beats its capitals, 5: 0.736The word in lower case beats its capitals, 6: 0.753The word in lower case beats its capitals, 7: 0.768The word in lower case beats its capitals, 8: 0.771The word in lower case beats its capitals, 9: 0.783The word in lower case beats its capitals, 10: 0.804The word in lower case beats its capitals, 11: 0.818The word in lower case beats its capitals, 12: 0.811The word in lower case beats its capitals, 13: 0.813The word in lower case beats its capitals, 14: 0.811The word in lower case beats its capitals, 15: 0.826The word in lower case beats its capitals, 16: 0.827The word in lower case beats its capitals, 17: 0.842The word in lower case beats its capitals, 18: 0.836The word in lower case beats its capitals, 19: 0.647The word in lower case beats its capitals, 20: 0.729The word in lower case beats its capitals, 21: 0.709The word in lower case beats its capitals, 22: 0.753The word in lower case beats its capitals, 23: 0.745The word in lower case beats its capitals, 24: 0.507The word in lower case beats its capitals, 25: 0.211The word in lower case beats its capitals, 26: 0.068The word in lower case beats its capitals, 27: 0.002Another word in lower case beats them, 0: 0.750Another word in lower case beats them, 1: 0.700Another word in lower case beats them, 2: 0.707Another word in lower case beats them, 3: 0.687Another word in lower case beats them, 4: 0.695Another word in lower case beats them, 5: 0.703Another word in lower case beats them, 6: 0.699Another word in lower case beats them, 7: 0.704Another word in lower case beats them, 8: 0.707Another word in lower case beats them, 9: 0.699Another word in lower case beats them, 10: 0.737Another word in lower case beats them, 11: 0.724Another word in lower case beats them, 12: 0.722Another word in lower case beats them, 13: 0.715Another word in lower case beats them, 14: 0.696Another word in lower case beats them, 15: 0.706Another word in lower case beats them, 16: 0.611Another word in lower case beats them, 17: 0.544Another word in lower case beats them, 18: 0.494Another word in lower case beats them, 19: 0.152Another word in lower case beats them, 20: 0.165Another word in lower case beats them, 21: 0.118Another word in lower case beats them, 22: 0.127Another word in lower case beats them, 23: 0.120Another word in lower case beats them, 24: 0.054Another word in lower case beats them, 25: 0.019Another word in lower case beats them, 26: 0.010Another word in lower case beats them, 27: 0.001

On held-out passages in capitals, the logit lens reads the coming word in lower case at the direction layer (0.753 of positions). The capitals form takes over for good only in the last three layers. Gemma-4-E2B shows the same shape.

Descriptive. Read across depth, the lens leans towards lower-case tokens in the first layers and shows the coming word only from mid depth: “lower case first” means that once the word can be read, it reads in lower case. From the paper’s Figure 4.

The numbers
The word in lower case beats its capitalsAnother word in lower case beats them
Layer 00.7910.750
Layer 10.7720.700
Layer 20.7660.707
Layer 30.6830.687
Layer 40.7080.695
Layer 50.7360.703
Layer 60.7530.699
Layer 70.7680.704
Layer 80.7710.707
Layer 90.7830.699
Layer 100.8040.737
Layer 110.8180.724
Layer 120.8110.722
Layer 130.8130.715
Layer 140.8110.696
Layer 150.8260.706
Layer 160.8270.611
Layer 170.8420.544
Layer 180.8360.494
Layer 190.6470.152
Layer 200.7290.165
Layer 210.7090.118
Layer 220.7530.127
Layer 230.7450.120
Layer 240.5070.054
Layer 250.2110.019
Layer 260.0680.010
Layer 270.0020.001
  • One direction switches the case. Adding it to a lower-case passage made the next word upper case at 0.981 of positions in Gemma-4-E2B and 0.956 in Qwen3-1.7B. Random directions of the same size flipped at most 0.003.
  • A few parts write and carry it, and the switch needs the MLPs after it. In Qwen3-1.7B, MLPs 19–22 and head L19H0 write it, and four heads carry the passage’s case. Freezing the MLPs after the direction layer stopped the flip: they are needed for it, which does not show they do nothing else.
Fig. 03 Which capitals use the direction: share still in capitals after it is removed
0.000.250.500.751.00Share still in capitalsQwen3-1.7BGemma-4-E2BAcronyms, Qwen3-1.7B: 1.000Acronyms, Gemma-4-E2B: 1.000AcronymsInitial capitals, Qwen3-1.7B: 0.968Initial capitals, Gemma-4-E2B: 0.982Initial capitalsHeadings in capitals, Qwen3-1.7B: 0.669Headings in capitals, Gemma-4-E2B: 0.768Headings in capitals0.000.250.500.751.00Share still in capitalsQwen3-1.7BGemma-4-E2BAcronyms, Qwen3-1.7B: 1.000Acronyms, Gemma-4-E2B: 1.000AcronymsInitial capitals, Qwen3-1.7B: 0.968Initial capitals, Gemma-4-E2B: 0.982Initial capitalsHeadings in capitals, Qwen3-1.7B: 0.669Headings in capitals, Gemma-4-E2B: 0.768Headings in capitals

Acronyms and words with an initial capital do not use it: removing the direction leaves them in capitals. Headings written in capitals do: removing it kept all capitals inside them at only 0.669 and 0.768.

From the paper’s Tables 3 and 23 (A1–A3).

The numbers
Qwen3-1.7BGemma-4-E2B
Acronyms1.0001.000
Initial capitals0.9680.982
Headings in capitals0.6690.768

Do the three share parts?

line-breaking indentation ordinal suffixes capitals counters, newlinewriters, readers block-end heads,keyword carriers MLP0 → MLP24,MLP18 writers, carriers,readers of the direction L17H6 · shared L11H9 · shared L17H6 also carries the passage’s case in capitals each behaviour’s own parts: mostly separate; two late heads have a small second role general early movers, used by several tasks L0H2moves the previous token L0H3moves the current token line-breaking indentation counters, newlinewriters, readers block-end heads,keyword carriers L17H6 · shared L11H9 · shared also carries the passage’s case in capitals capitals ordinal suffixes writers, carriers andlate MLP readers MLP0 → MLP24,MLP18 each behaviour’s own parts: mostly separate general early movers, used by several tasks L0H2moves the previous token L0H3moves the current token

The cross-study test in Qwen3-0.6B. Capitals were tested in Qwen3-1.7B, with heads matched by index; the line-breaking parts come from the companion paper on line-breaking.

A separate frozen test asked whether the groups each study named are really the same parts. It found less sharing than earlier reports of component reuse:

  • The groups that do most of each task’s work are separate. Switching them off harms the other tasks no more than random groups of the same layers do.
  • Two late heads have a small second role. Without L17H6, the model kept only 0.706 of the effect of line width on where it breaks a line, against 0.988 to 1.036 for random heads of its layer, and the head adds a little to the dedent after return. L11H9 helps choose the ordinal suffix and adds to the dedent after a terminal statement: a role first noticed after the cross-study test, then confirmed by a frozen test on new material (−0.027, below all twenty random heads of its layer).
  • General early movers. The early heads the studies named, L0H2 and L0H3, move the previous or the current token in every task: general parts, not task parts.
  • What “separate” means here: within bounds fixed in advance, on these measures, at the decision positions. Comparing every value with twenty layer-matched random groups is post hoc, except for L11H9’s second role, which the third round tested prospectively. Sharing through MLPs and the residual stream is not ruled out.

How it was tested

  1. 01

    Gate

    A model was studied only if it did the task well. Pythia-410M and GPT-2 failed the indentation gate; Qwen3-0.6B failed the capitals gate.

  2. 02

    Explore

    Under a frozen charter: ablation, patching, attention, the logit lens and steering, until a candidate mechanism appeared.

  3. 03

    Freeze

    Predictions, thresholds, controls and code written down and frozen. Every named set is compared with twenty random sets from the same layers.

  4. 04

    Test once

    On material sealed before any model read it, two rounds per study. A third round re-ran every code-based test on code that can be shared.

The predictions were written after exploration, so the counts below record that each account repeated on held-out material, not that it foresaw effects nobody had looked for. A test is one prediction read in one model (and, for ordinals, at one number length).

  • 9 of 9indentation, round 1
  • 25 of 25indentation, round 2
  • 34 of 34indentation, round 3, on shareable code
  • 14 of 14ordinals, round 1
  • 39 of 41ordinals, round 2
  • 19 of 19capitals, round 1
  • 23 of 23capitals, round 2
  • 22 of 22shared parts
  • 15 of 15shared parts, round 3

The two failed tests are one ordinal prediction at its two number lengths, in Qwen3-1.7B: that its early previous-token heads would leave the last digit’s pull intact. They removed much of it too (0.548 and 0.233 kept, against the 0.55 required). In that model the two kinds of early head divide the work less cleanly.

What this doesn’t show

  • Small base models (the largest, Gemma-4-E2B, has 2.3 billion effective parameters), English text, Python only for indentation, numbers up to ten digits, and Wikipedia paragraphs and headings for capitals. The ordinal path was traced only in Qwen3-0.6B.
  • The accounts are partial. What tells the block-end head where to look is spread over many components, and the ordinal signal between MLP0 and MLP24 is spread thinly and untraced. Removing the capitals direction left about half of the words in capitals, so case is also carried in other ways.
  • Mean ablation, direct attribution, the logit lens and patching choices all have known weaknesses. Random-set controls bound these concerns but do not remove them.
  • The cross-study test used one model family, with capitals heads matched by index, and ablated at the decision positions only. The third round re-ran only the code-based tests.
  • Held-out material is held out from these analyses, not necessarily from the models’ training data. The code and the Wikipedia articles come from repositories and pages created after the models were trained, but these can contain older code and text.

How the study went

  1. For each study: a frozen charter, exploration, then two frozen rounds on sealed held-out material, the second tracing the first round’s account into its parts. Then a frozen cross-study test of shared parts.

  2. A measurement error that dropped points after blank lines was found and fixed by a dated amendment; it changed no gate verdict. A post hoc weight comparison showed that OLMo-2’s published checkpoints are not ancestors of the released model, and the inference that depended on it was withdrawn.

  3. A third indentation round, frozen before any model read its material, re-ran every code-based test on permissively licensed and newly generated code. All held again, and it confirmed L11H9’s second role, first noticed after the cross-study test.

  4. This version, with the code and data archive: 34 pages.

Read it, cite it

Status
Working paper, not peer-reviewed
This version
12 November 2026
Field
Language models
Models
Qwen3-0.6B, Qwen3-1.7B, OLMo-2-1B, Gemma-4-E2B, and others
Archive
doi:10.5281/zenodo.24150049
Read the abstract

Small language models make many small formatting and morphology decisions correctly: how far to indent the next line of Python, whether a number takes st or th, whether the next word continues a passage written in capitals. We asked how they make three of these decisions, and then whether the mechanisms share parts. Each study explored under a frozen charter, then wrote down what the explored account predicts and tested those predictions, with thresholds and layer-matched random controls, once on held-out material, so the tests show whether each account repeats on new material. Indentation (Qwen3-0.6B-Base and OLMo-2-0425-1B): dedents reach the model by two routes. After return and similar statements the keyword is read directly from the decision token through many heads. Unmarked dedents depend on a small set of late heads, chief among them one that attends to the line-ending token of an open block’s header and copies that header’s indent; which open header it attends to sets how far to dedent, and steering its attention moves the prediction. Ordinal suffixes (Qwen3-0.6B-Base, OLMo-2-0425-1B and five further models): to write “112th”, Qwen3-0.6B-Base uses an early previous-token head to bring the tens digit to the last digit; the first MLP reads it, and a late MLP writes a teen override towards th that does not consult the last digit, while a different MLP writes the last digit’s own suffix. In OLMo-2-0425-1B, where the tokeniser can put the tens digit in another token, the signal arrives too faint, and a thousands separator repairs the errors; early previous-token heads carry part of the check in three further models. In the ordinal study’s second round two tests failed, the two number lengths of one prediction. Capitals (Qwen3-1.7B-Base and Gemma-4-E2B): the next word is chosen in a lower-case-like form and turned into capitals in the last layers along one direction, which is written by a few late components and fed by heads that carry the passage’s case; the switch to capitals needs the MLPs after it. A prospective cross-study test (line-breaking, indentation and ordinals in Qwen3-0.6B-Base; capitals in Qwen3-1.7B-Base, with heads matched by index) finds less sharing than earlier reports of component reuse. The early heads the studies named are general movers of the previous or the current token. Of the later heads, two have a small second role: L17H6 is shared between line-breaking and indentation, and L11H9 serves both the ordinal suffix choice and the dedent after a terminal statement, a role first noticed after the cross-study test and then confirmed by a frozen test on new material. The groups that do most of each task’s work harm the other tasks no more than random groups do. A third round re-ran every code-based test of the indentation and cross-study work on permissively licensed and newly generated code, which can be redistributed, and all of them held again.

BibTeX

@misc{jeevash2026small,
  title  = {{Small decisions, mostly separate parts: how small language models indent code, choose ordinal suffixes and write in capitals}},
  author = {Jeevash, Richard},
  year   = {2026},
  note   = {Working paper, not peer-reviewed. Code and data: doi:10.5281/zenodo.24150049},
  url    = {https://richardjeevash.com/research-small-decisions.html}
}