Your LLM re-reads the same text a thousand times

I spent two weeks trying to make CPU inference faster. The thing that finally worked was not in the engine.

Measured on a free Oracle ARM box — 4 cores, €0/month. Everything below is reproducible; the scripts are linked at the end.

I maintain a CPU-first inference server built on llama.cpp. Not as a fallback for a missing GPU — as the target. The hardware I care about is the free tier: two to four ARM cores, no accelerator, no budget.

The bottleneck there is prefill. Before a model writes a single token it has to read your entire prompt, and on a document of any size that read dominates everything else. So I went looking for speed in the obvious places.

Three things that didn't work

I bumped llama.cpp forward by a year of upstream kernel work, expecting the ARM optimizations to show up. On this quantization and this CPU: 0%.

I tried a coarser quantization. Q4_0 is 37% faster at prefill and 19% faster at decode — and it dropped 5 facts out of 20 on my extraction test. Rejected.

I halved the number of active experts during prefill on a mixture-of-experts model. 44% faster, and it silently corrupts the cache: a KV cache built with 4 experts and read back with 8 scores 11/20 against a 14/20 control. The damage is baked into the cached representation, not just the output. That one was worth knowing about.

What worked was the text

The same page. The same model. The same question. Rewritten as label: value, one fact per line, with conditions kept attached to what they qualify:

Six and a half times faster, and more accurate. That second part is what I didn't expect, and it's the reason this is worth writing down. Fewer tokens should mean less information and worse answers. It didn't.

The model saw the right number and answered wrong

To understand why accuracy went up, I dumped the post-softmax attention while the model answered a question about a page listing four similar prices — one per service, all in the same block.

1.9% against 1.7% is a coin flip. The model was never blind to the number — it just couldn't bind it to the question. Prose puts the price and the service in the same sentence as everything else; label: value makes the binding structural instead of semantic, and the ratio goes from 1.1:1 to 7:1.

The practical consequence: when a small model gets a fact wrong, "use a bigger model" is one answer, but "write the document differently" is often cheaper and works better.

Then I went looking for where the bytes go

Having stopped guessing, I counted what a model actually reads per generated token. Two things came out that I have not seen written down anywhere, both invisible on a GPU.

1. The output head is a quarter of your bandwidth

In many small models the embedding matrix is tied: it doubles as the output projection and is read in full for every token generated, to score all 151,936 vocabulary entries.

Nearly a third of the bandwidth per token goes to scoring words in languages this deployment will never emit. My domain — Italian, my own code, technical English — uses 9,314 of those 151,936 entries.

Trimming the vocabulary to 32k turns out to be safe by construction, for two reasons worth stating precisely. The tokenizer is byte-level and all 256 byte-characters are kept, so no text becomes unrepresentable — the worst case is that a trimmed word costs an extra token. And the embedding is Q6_K with rows spanning a whole number of quantization blocks, so whole rows drop out without ever splitting a block: the surviving weights are bit-for-bit identical. No requantization, no numerical loss.

Measured on the free box, three independent runs, under 2% spread: +17.8% decode, from 46.7 to 55.0 tokens/second. The 20-question fact exam scores the same before and after — 13/13 critical facts both times. The cost, measured on held-out text, is 1.9% more tokens for the same document, so this pays off most when the input is already short.

The theoretical ceiling from bandwidth alone was +30%. I got +17.8%. The gap is time that isn't weight reading, and I'd rather publish the measured number than the calculated one.

2. Weight repacking costs 4.2 GB nobody mentions

My server showed 9,825 MB resident for a 5.37 GB model. I assumed a leak. It isn't:

load_tensors: CPU_Mapped model buffer size = 5068.51 MiB
load_tensors: CPU_REPACK model buffer size = 4254.45 MiB

llama.cpp repacks quantized weights at load time into a layout its NEON kernels read faster. It's a sensible engineering trade — and on a GPU it doesn't exist, which is probably why the memory cost isn't documented anywhere I could find.

I had first read the source and concluded the repack only touched 2D tensors, which would have spared the MoE experts. The logs said otherwise: ffn_gate_exps and ffn_up_exps get repacked too, and ffn_down_exps goes from q6_K to q8_0_4x4 — from 6 bits to 8. The repack doesn't only reorganize; in places it expands.

−4.2 GB of RAM for −12.8% speed. On a machine with memory to spare that's a bad trade. On an 8 GB box it's the difference between the model running at all and not running — which is exactly the hardware this is for.

What these numbers don't show

The fact exam is mine: 20 questions I wrote over one real Italian business page, graded by regex. One page, one language, one domain. It is the weakest part of this write-up and I'd rather say so than have it pointed out.

What keeps it honest is that the same exam killed two of my own ideas — the faster quantization and the reduced experts, both above. And on that exam the fast MoE I serve is beaten by a slower dense 3B. Reformatting the input recovered accuracy the fast model was losing; it did not make the small model smart. Asked anything with no document supplied, it invents things — I published that table too.

If you know a public adversarial fact-extraction set for small models, point me at it and I'll run it and publish whatever comes out, including a bad result.

The through-line

Every win here came from measuring where the work actually goes, and every loss came from assuming. The kernel bump, the coarser quantization, the reduced experts: all reasonable ideas, all wrong on this hardware. The text rewrite, the vocabulary trim, the repack switch: all boring, all measurable, all significant.

On a CPU, the fastest computation is the one you don't do — whether that's a token you never read, a vocabulary entry you never score, or a byte you never repack.