Reinforcement Learning in Large Language Models: Rewards, Credit, and Exploration

Reinforcement learning (RL) is one way to train a language model when the useful feedback is a reward, preference, or test result instead of a single target sequence. The reward may be produced by a rule-based checker, a learned reward model, or human preferences. Each source changes what the resulting policy is optimized to do.

This article compares PPO, RLHF, and GRPO, explains how sequence-level rewards are applied to token-level updates, and describes why supervised fine-tuning (SFT) is often used before RL. Exploration can expose a policy to more than one response, but it does not by itself guarantee better generalization or safer behavior.

1. Different RL methods

Many policy-gradient methods update model parameters in the direction that increases an objective. A compact notation is:

theta <- theta + alpha * gradient_theta J(theta)

Here theta is the vector of model parameters, alpha is a learning rate, and J(theta) is the objective being optimized. The expression is a teaching abstraction. The estimator for the gradient, the data used for it, and the constraints added to the objective differ between algorithms.

PPO

Proximal Policy Optimization (PPO) compares the probability of a sampled action under the current policy with its probability under the policy that generated the sample. For a token or action at step t, the ratio is:

r_t(theta) = pi_theta(a_t | s_t) / pi_old(a_t | s_t)

PPO clips this ratio in its surrogate objective. With A_t as an advantage estimate and epsilon as the clipping range, the central term is:

min(r_t(theta) * A_t,
    clip(r_t(theta), 1 - epsilon, 1 + epsilon) * A_t)

The clipping rule limits how much a sampled update contributes to this objective. A KL penalty may add another constraint against a reference policy. These mechanisms shape the update, but they do not guarantee stable training, factual outputs, or safe behavior in every setting.

RLHF

Reinforcement Learning from Human Feedback (RLHF) is a training recipe, not a single optimizer. A typical recipe collects preference comparisons, trains a reward model to predict those preferences, and then optimizes the language model against that learned reward. PPO is one possible optimization method in the final stage. A KL term is often included to limit drift from a reference model.

The reward model is an approximation of the annotators and data used to train it. It can reward superficial features or other behavior that scores well without satisfying the intended goal. This is one form of reward hacking. A rule-based checker can reduce ambiguity for tasks with an executable specification, but it is only as complete as the specification.

GRPO

Group Relative Policy Optimization (GRPO), used in the DeepSeek-R1 work, samples several responses for one prompt and estimates each response’s advantage relative to the rewards in that group. It does not train a separate value-function critic in the PPO style. This changes the memory and estimation tradeoffs; it does not make the reward objective universally reliable.

2. Why the best answer alone is not the whole SFT objective

SFT trains the model to predict target tokens from input examples. It can use discrete target tokens because the cross-entropy loss is differentiable with respect to the model parameters. The target itself does not need to be differentiable. For gradient-based neural-network training:

Supervised learning can use discrete targets; the training loss must be differentiable with respect to the model parameters.

Training only on one selected answer can narrow the examples seen by the model. If several answers are valid, a dataset can include multiple demonstrations or a rubric that captures the shared requirements. SFT remains useful because it gives the policy a distribution of examples from which later sampling can begin.

Sequence rewards and token-level signals

For many language tasks, a checker or preference model scores the completed response. Let R(y) be the scalar reward for sequence y, and let a_t be the token sampled at position t. A simple policy-gradient contribution has the form:

gradient_theta E[R(y)]
  ≈ R(y) * sum_t gradient_theta log pi_theta(a_t | a_<t, x)

The sequence reward is attached to the log-probability of each sampled token. Advantage estimates, baselines, and clipping can change the weight for each position. This is a way to train token probabilities from a sequence-level signal; it is not proof that every token caused the final score.

SFT applies a target-token loss at positions for which the dataset supplies labels. RL can instead use a scalar result that is not differentiable through the environment, such as “the program passed” or “the answer won.” The policy loss remains differentiable with respect to model parameters even when the reward calculation is not.

Sampling and exploration

RL training often samples several candidate responses and reinforces those with higher reward. This can expose the update to alternatives that are absent from a single demonstration. What it learns depends on the prompts, sampling distribution, reward coverage, and optimization settings. Exploration may help a task’s measured score, but it does not guarantee improved performance on unseen inputs or protection against reward hacking.

3. SFT followed by RL in DeepSeek-R1

The DeepSeek-R1 paper illustrates two related pipelines. R1-Zero starts from DeepSeek-V3-Base and applies GRPO without a supervised fine-tuning stage. For the reasoning tasks in that experiment, rule-based rewards evaluate answer correctness and required format. The paper reports R1-Zero’s AIME 2024 pass@1 rising from 15.6 to 77.9 during training, with the exact value depending on the reported checkpoint and evaluation setup.

The full R1 pipeline adds a cold-start SFT stage, reasoning-focused RL, rejection sampling followed by more SFT, and a second RL stage. Its reasoning rewards include verifiable signals for domains such as mathematics and coding. For general data, the paper also describes model-based rewards. “No LLM as reward” is therefore a statement about the appropriate rule-based reasoning stages, not a description of every reward in full R1.

SFT can provide examples of useful formats and a starting policy. RL can then optimize an explicit reward over sampled responses. The combination is a design choice whose value must be measured against the chosen base model, data, reward, and evaluation protocol. SFT followed by RL is not a general guarantee of alignment or performance.

4. What exploration can and cannot tell us

Sampling multiple responses creates a broader set of trajectories than copying one target. A group-relative method such as GRPO can learn which sampled outcomes score better than the group reference. A reward model or verifier then determines which behavior is reinforced.

That process can improve a benchmark when the reward tracks the benchmark’s objective. It can also reinforce a shortcut when the reward is incomplete. Generalization depends on the relationship between training prompts, evaluation prompts, model capacity, and reward coverage. An apparent improvement on an in-distribution test does not by itself establish out-of-distribution robustness.

KL penalties and clipped objectives are useful controls on the optimization path. They constrain parts of the update relative to a reference policy, but they do not ensure that the reference is correct or that unsafe behaviors cannot be learned. Evaluation still needs task-specific correctness checks, preference audits, and tests outside the data used for optimization.

Conclusion

PPO is an optimization algorithm, RLHF is a broader preference-training recipe, and GRPO estimates relative advantages from groups of sampled responses without a separate PPO-style critic. SFT supplies labelled examples, while RL can optimize rewards that arrive only after a whole response is complete.

The important engineering question is not whether RL is always better than SFT. It is whether the reward, sampling procedure, update constraints, and evaluation set measure the behavior the system is meant to produce. DeepSeek-R1 shows one high-profile SFT and RL pipeline, while the distinction between rule-based verification and learned reward models explains why its stages use different feedback sources.