DeepSeek-R1 and Reinforcement Learning for Reasoning

DeepSeek’s DeepSeek-R1 paper describes two related systems. DeepSeek-R1-Zero is an experiment in reinforcement learning directly from a DeepSeek-V3 base model. DeepSeek-R1 adds supervised data, rejection sampling, and additional reward signals to make the resulting model more readable and useful across reasoning and general tasks.

DeepSeek-R1-Zero: reinforcement learning first

R1-Zero starts from DeepSeek-V3-Base and skips supervised fine-tuning. For each question, Group Relative Policy Optimization (GRPO) samples a group of outputs. Rule-based rewards check the final answer for mathematical, coding, and logical reasoning tasks, and a format reward checks the required output structure.

The rule-based reward applies to those verifiable reasoning stages. It is not a statement that the complete R1 system never uses model-based rewards. R1-Zero’s training used 16 sampled outputs per question, continued for 10,400 steps, and reached 1.6 training epochs in the paper’s reported setup.

During training, the paper reports AIME 2024 evaluation pass@1 increasing from 15.6% to 77.9%. Self-consistency decoding with 16 samples, reported as Cons@16, reaches 86.7%. These are measurements of the R1-Zero training trajectory and its stated decoding setting.

R1: a multi-stage pipeline

R1 addresses R1-Zero’s language mixing and readability issues with additional stages:

1. Cold-start SFT uses curated reasoning examples.
2. Reasoning-focused reinforcement learning uses the verifiable reasoning rewards described above.
3. Rejection sampling and SFT combine accepted reasoning traces with non-reasoning data, including writing and other instruction tasks.
4. A second RL stage improves general helpfulness, harmlessness, and language consistency.

The final RL stage uses different signals for different data. For reasoning data, the paper uses rule-based rewards. For general data, it uses reward models and format rewards, together with a language-consistency reward. “No LLM as reward” is therefore a description of the rule-based reasoning stage, not of every R1 training objective.

GRPO and PPO

GRPO is related to PPO but changes how the advantage is estimated. For each question, GRPO compares rewards within a group of sampled outputs and normalizes each output against that group. It does not train a separate value model or critic for the advantage estimate.

PPO commonly learns a value function, or critic, to estimate expected return and uses that estimate in its policy update. These are different algorithmic choices. Describing GRPO as “PPO without a critic” is a useful short comparison, but it does not make all reinforcement-learning methods equivalent.

Reported results

The paper’s final R1 scores use different metrics for each benchmark. Selected rows are reproduced below so the model, benchmark, and metric stay visible:

Values are from the final R1 column in Table 3 of the DeepSeek-R1 paper. Pass@1, percentage, rating, resolution, and win-rate are different measurements and should not be combined into one score.

The paper also releases distilled R1 models based on smaller open models. Their results depend on the particular distilled model, task, and decoding setup, so the existence of a distilled checkpoint does not establish general capability parity with a larger model.

The distinction between R1-Zero and R1 matters when interpreting these numbers. R1-Zero isolates direct reinforcement learning on verifiable reasoning, while R1 combines that reasoning work with supervised data, rejection sampling, and model-based rewards for general data.