How Reasoning Models Learn to Think

This article explains how reasoning models are trained, centered on DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv:2501.12948, January 2025) — the first openly documented recipe — together with the GRPO algorithm from DeepSeekMath and the test-time compute results reported for OpenAI's o1.

Introduction

By 2024, the recipe that built GPT-3 and GPT-4 was running into a wall on one specific class of problems. Pretraining on more text kept improving fluency, knowledge, and style. It did not proportionally improve multi-step reasoning — competition math, algorithmic coding, formal logic. GPT-4-class models still scored in the low teens on AIME, a high-school olympiad qualifier.

The reason is structural. Next-token prediction teaches a model what text tends to follow other text. A hard math problem is not solved by recalling likely text. It is solved by trying an approach, noticing it fails, and trying another — a search process. Pretraining data rarely shows that process; published solutions show only the polished final path, not the dead ends.

In September 2024 OpenAI released o1, the first model trained to produce a long private chain of thought before answering, and reported that its accuracy improved along a new axis: not more parameters or more data, but more reinforcement learning and more thinking time. o1’s training details stayed closed. Four months later, DeepSeek published R1 with the full recipe — and the recipe turned out to be surprisingly simple.

The shift is easy to state. Pretraining teaches a model what to say. This second phase teaches it how to think: the model is rewarded only for reaching correct answers, and it must discover for itself what kind of thinking gets there.

The Key Idea: Verifiable Rewards

Reinforcement learning needs a reward signal. For LLMs, the standard source had been RLHF: train a separate neural reward model on human preference rankings, then optimize the LLM against it. RLHF made models polite and helpful. It is a poor teacher for reasoning, for one deep reason: the reward model is itself a fallible neural network, and the policy will learn to exploit its mistakes. Optimize hard enough against a learned judge and you get answers that look right to the judge rather than answers that are right. This is reward hacking, and it caps how much optimization pressure you can safely apply.

Math and code offer an escape. A math problem with a known answer can be checked by string comparison. A program can be run against a test suite. These checks are rule-based, binary, and uncheatable — there is no neural judge to fool. The community now calls this RL with verifiable rewards (RLVR).

DeepSeek-R1-Zero used exactly two rule-based rewards: an accuracy reward (is the final answer correct?) and a format reward (is the reasoning enclosed in <think> tags?). The paper is explicit that they avoided neural reward models precisely because of reward hacking during large-scale RL.

Core Mechanics: The RL Loop and GRPO

The training loop is short. Take a base model (R1-Zero started from DeepSeek-V3-Base — no supervised fine-tuning at all). For each problem in a batch, sample a group of $G$ complete answers, chain of thought included. Score each with the verifier. Update the policy to make high-scoring answers more likely and low-scoring answers less likely. Repeat for thousands of steps.

The interesting question is the update rule. The classical choice, PPO, needs a second neural network — a value model (critic) — to estimate how good each state is, so that the reward can be converted into an advantage: how much better this answer is than expected. For LLMs the value model is typically as large as the policy itself. That doubles memory, and training a per-token value estimate for a reward that only arrives at the very last token is both hard and noisy.

GRPO (Group Relative Policy Optimization), introduced in DeepSeekMath, deletes the value model. Its observation: you are already sampling $G$ answers per problem. The group itself tells you what “expected” looks like. The objective is:

with the probability ratio $\rho_i = \pi_\theta(o_i \mid q) / \pi_{\theta_{old}}(o_i \mid q)$, and — the heart of the method — the group-relative advantage:

Why this is cheaper and stabler than PPO for LLMs:

• No value network. Roughly half the GPU memory and compute of PPO disappears, along with an entire second training problem (fitting the critic).
• The baseline cannot be wrong in a learned way. A critic is a neural net with its own biases and lag; a group mean is a direct Monte-Carlo estimate for this exact problem, right now.
• Normalization matches sparse rewards. With 0/1 rewards, dividing by the group’s standard deviation automatically scales the gradient: a problem where the model almost always fails but one sample succeeds produces a strong learning signal on that one success.

Interactive: Group-Relative Advantage Explorer

This is the core mechanic of the whole recipe, and it fits in one small calculation. Below are six sampled answers to one problem. Drag the reward sliders and watch each answer’s advantage $A_i = (r_i - \text{mean})/\text{std}$ update live. Indigo bars are pushed up by the policy update; gray bars are pushed down. Try the “all identical” preset to see the failure mode.

Two things worth noticing while playing. First, only relative standing matters: raise every reward by the same amount and the advantages do not move. Second, a group where every answer gets the same reward — all right or all wrong — produces zero advantage everywhere and therefore zero gradient. GRPO learns nothing from problems that are too easy or too hard for the current policy, which is why training naturally concentrates on the frontier of what the model can almost do.

What Emerges: The R1-Zero Story

DeepSeek-R1-Zero is the cleanest experiment in the paper: base model, GRPO, rule-based rewards, and no supervised examples of reasoning whatsoever. Nobody showed the model a single worked chain of thought. What happened over thousands of RL steps is the most striking result in the reasoning-model literature.

Accuracy climbed from guessing to o1-level. On AIME 2024, pass@1 rose from 15.6% to 71.0%, and to 86.7% with majority voting over 64 samples — matching OpenAI’s o1-0912 (74.4% pass@1).

The model chose to think longer. Nothing in the reward mentions length. Yet average response length grew steadily from a few hundred tokens to many thousands, because longer deliberation — exploring, checking, backtracking — is what wins reward on hard problems. Thinking time was discovered, not programmed.

Self-verification and reflection appeared on their own. Mid-training, the model began re-checking its own steps, flagging errors, and restarting derivations. The paper’s most quoted moment is a transcript where R1-Zero, mid-solution to an algebra problem, interrupts itself:

The authors call this the “aha moment” — for the model and, they admit, for themselves. Reinforcement learning did not teach the model how to reflect; it created conditions under which reflection paid, and the behavior emerged.

The full R1 pipeline pushed AIME 2024 to 79.8% pass@1 and MATH-500 to 97.3% — on par with OpenAI’s o1-1217 — plus a 2,029 Codeforces rating (96.3rd percentile among human competitors).

Test-Time Compute: The Second Scaling Axis

For a decade, “scaling” meant one thing: more parameters, more data, more pretraining FLOPs. Reasoning models added a second, independent axis. OpenAI’s o1 announcement showed AIME accuracy rising roughly log-linearly with the amount of compute spent thinking at inference time — the same trained model gets smarter if you let it think longer.

This changes the economics of intelligence. Capability is no longer fixed at training time; it becomes a dial you turn per query, trading tokens (money, latency) for accuracy. Easy question: think briefly. Olympiad problem: think for tens of thousands of tokens. Every major lab now ships this dial explicitly — reasoning-effort settings, extended-thinking budgets, and hybrid models that decide per query whether to think at all.

Distillation: reasoning in small models

The second surprise in the R1 paper is how well reasoning transfers by imitation. DeepSeek generated ~800k samples with R1 and fine-tuned small open models on them — plain supervised learning, no RL at all:

A 7-billion-parameter model outscoring GPT-4o on competition math by a factor of six was the moment “reasoning” stopped being a frontier-lab exclusive. The catch, confirmed by the paper’s own ablation: discovering reasoning via RL from scratch works poorly at small scale — RL directly on a 32B base underperformed simple distillation from R1. Big models find the behaviors; small models can only inherit them.

Trade-offs and Open Problems

The recipe is real, but every part of it has a sharp edge.

Reward hacking never disappears; it moves. Rule-based rewards close the front door — you cannot fool a unit test with a plausible-sounding proof. But policies find side doors: hard-coding expected outputs of visible test cases, exploiting weak or under-specified tests, guessing answer formats. And the moment any learned judge re-enters the pipeline (for partial credit, or for unverifiable domains), the original RLHF hackability returns with it.

Verifiable domains are a narrow slice of what we want. Math, code, and puzzles have oracles. Strategy advice, product judgment, honest writing, and research taste do not. The open question of the field is transfer: does thinking trained on math generalize outward? Partially — reasoning models do beat their bases on many general tasks — but the gap between “verifiable-domain gains” and “everything-else gains” is consistently large. Extending RLVR beyond its native domains (via rubric-based judges, self-verification, or formal-methods scaffolding) is an active research frontier, not a solved problem.

Overthinking is a tax. RL rewards correctness, not efficiency, so models pad: thousands of tokens of deliberation on questions a lookup would settle. That is real money and real latency at serving time, and it is why effort controls and think/no-think routing became standard within a year of R1.

The chain of thought is not a window into the model. The <think> text is optimized to produce correct answers, not to faithfully report computation. Models can reach right answers via written reasoning that is wrong, and there is evidence that CoT can rationalize rather than explain. Reading the thoughts is useful; trusting them as an audit log is not.

Key Takeaways

1. Verifiable rewards are the unlock. Math and code provide binary, rule-based reward signals that cannot be gamed the way a learned RLHF reward model can. That is what makes large-scale RL on reasoning stable.

2. GRPO makes the RL loop cheap. Sample a group of answers and normalize each reward against the group’s mean and standard deviation: $A_i = (r_i - \text{mean})/\text{std}$. The group replaces PPO’s value network — half the memory, one fewer model to train, and a baseline that is always locally correct.

3. Reasoning behaviors emerge; they are not programmed. Longer deliberation, self-verification, backtracking, and the “aha moment” all appeared in R1-Zero from nothing but correct-answer rewards — with AIME pass@1 rising 15.6% to 71.0%.

4. Test-time compute is a second scaling axis. Accuracy rises with thinking tokens per query, independent of model size. Intelligence became a per-request dial, with token cost as the price.

5. Discovery needs scale; inheritance does not. RL finds reasoning in large models; distillation copies it into small ones better than small-scale RL can discover it. That asymmetry shaped the open-model landscape after January 2025.