Vision Transformers Need Registers

This article explains the ICLR 2024 paper Vision Transformers Need Registers by Darcet et al. Figures marked "from the paper" are the authors' own; the probing chart and the interactive are drawn for this article from the paper's numbers and attention maps.

A speck of sky that will not stay quiet

Ask a Vision Transformer where it is looking, and you expect its attention to settle on the object. The usual way to ask is to plot the attention of [CLS], the extra token whose output summarizes the image. For the original DINO model, that is what happens. For the big modern ViTs that everyone builds on, something else shows up too: a handful of bright specks scattered over plain background, far from anything interesting.

Input

DINOv2

DINOv2 + registers

This is not one model’s quirk. The specks appear in DeiT-III (trained on labels), OpenCLIP (trained on image-text pairs) and DINOv2 (self-supervised). Three different training methods, the same symptom.

The paper’s answer, and the idea that holds this whole post together: the specks are the model making its own scratch space. It needs somewhere to keep information about the whole image while it computes, it has no slot for that, so it takes over patches it thinks nobody will miss. The fix is to give it the slots.

Three clues about the specks

Before the explanation, the evidence. The authors characterize the specks along three lines, and each one points the same way.

Clue 1: they are loud

Each output token is a vector. Measure its length (its norm) for every patch across many images, and the patches split into two groups. Most sit in a tight band. A small group has norms many times larger. In DINOv2 ViT-g, about 2.4% of patch tokens exceed a norm of 150; the bright specks in Figure 1 are exactly these high-norm tokens.

A population that splits cleanly in two usually means two different jobs, not one noisy job.

Clue 2: they sit on boring patches

Where do the high-norm tokens appear? The authors compare each patch with its neighbours in the input. The high-norm patches are almost identical to the patches around them: a stretch of sky, a smooth wall, an out-of-focus background.

In other words, the model picks the patches that are redundant. Losing what they say costs almost nothing, because their neighbours say the same thing.

Clue 3: they appear only when the model is big and well trained

The specks are not there from the start.

A behavior that needs depth, training time and capacity is not a bug in the data or the code. It is something the model learns to do once it is strong enough to benefit from it.

What the specks are doing

So the model deliberately overwrites a few redundant patches. With what? The authors answer with linear probes: small linear classifiers trained on frozen token vectors, which test what information a token still holds. They probe normal patches and high-norm patches for two kinds of information.

Read the two panels as a trade. The high-norm tokens got worse at their own job: knowing where they are and what pixels they cover. They got better at a job that was never theirs: describing the whole image. The model has erased a local patch and written a global summary in its place.

Try it: find the hijacked patches

The maps below are the real attention maps from the paper, at patch resolution. The circles mark patches that are far brighter than their surroundings and that go dark in the model trained with registers, detected automatically from the two maps. Switch to the model trained with registers and watch where the global information goes.

Where does the global summary go?

ModelDINOv2DeiT-III

Registers04

Register tokens

reg 1

reg 2

reg 3

reg 4

DeiT-III shows the problem more strongly than DINOv2. Its specks are larger and there are more of them, which is also why methods that read its attention maps struggled most, as the results below show.

The fix: give the model registers

If the model is improvising scratch space, hand it some. The paper adds a few extra learnable tokens, called registers, to the input sequence.

A register has no pixels behind it and no output job. It starts as a learned vector, is updated by attention like every other token, and is thrown away at the end. Its only purpose is to give attention heads a legitimate place to read and write image-wide information. The recipe is three steps:

1. Append $N$ learnable tokens to the input, next to [CLS].
2. Train as usual. Registers join every attention operation.
3. Discard them at the output. Downstream tasks use [CLS] and the patch tokens as before.

The cost is small: with 4 registers, compute grows by under 2%; with 16, by up to 6%. The real cost is elsewhere: the paper’s models are trained from scratch with registers, because the hijacking habit is learned in pretraining.

How many registers?

A single register already removes the artifacts. Beyond that, the tasks disagree: segmentation peaks around 4 registers, depth around 8, and ImageNet keeps improving up to 16. The paper uses 4 as its default, a middle ground at under 2% extra compute.

What changes with registers

The loud tokens go quiet

Look back at Figure 2, right column of each pair. With registers, the high-norm tail shrinks sharply in all three models, and for DINOv2 and CLIP it vanishes into a single band. The specks in Figure 1 disappear with it.

Dense tasks: small, mostly positive changes

Linear-probe results from the paper. mIoU is the overlap between predicted and true segmentation masks (higher is better); RMSE is the depth error (lower is better). The gains are modest, largest for DINOv2, and not universal: DeiT-III’s depth error barely moves. Registers are mainly a fix for the feature maps, not a general accuracy boost.

Object discovery: the big change

The clearest win is for methods that read the attention and feature maps directly. LOST finds the main object in an image without any labels, by starting from the least-connected patch and growing outward. A bright speck on the background can hijack that seed.

The metric is corloc: the share of images where the predicted box overlaps a real object enough to count. For DINOv2, object discovery jumps by 15 to 20 points on every benchmark, and DeiT-III more than doubles. OpenCLIP is the exception and gets slightly worse, a reminder that registers fix one specific failure rather than everything.

What the registers learn

Nobody tells the registers what to do. Still, their attention patterns come out different from each other, and some settle on particular objects in the scene.

Input

[CLS]

Register 0

Register 6

Register 8

The same story in language models

The pattern is not unique to vision.

A 2025 follow-up, Vision Transformers Don’t Need Trained Registers (Jiang et al.), removes the main cost. It traces the high-norm tokens to a small set of neurons and redirects their activity into an extra, untrained token at test time. That mimics registers on models already trained without them, with no retraining.

Takeaways

1. The specks are scratch space. Large ViTs overwrite a few redundant background patches with image-wide information. Those tokens become high-norm outliers and show up as bright specks.

2. Three clues point to it. The tokens are loud (norms many times larger), they sit on patches that copy their neighbours, and they appear only in big models, mid-network, after a third of training.

3. Probes confirm the trade. High-norm tokens forget where they are (22.8% vs 41.7% position accuracy) and know more about the whole image (69.0% vs 65.8%).

4. Registers are the fix. A few learnable tokens that are discarded at the output give the model legitimate scratch space. One removes the artifacts; four is the default, at under 2% extra compute.

5. The payoff is clean feature maps. Dense-task gains are modest, but methods that read attention maps directly, like LOST, improve by 15 to 20 points on DINOv2.