REFT: Reasoning with Reinforced Fine-Tuning

REFT: Reasoning with REinforced Fine-Tuning, by Trung Quoc Luong and co-authors at ByteDance Research, studies a two-stage method for mathematical reasoning. It uses supervised fine-tuning (SFT) to warm up a model, then samples new reasoning paths online and optimizes them with Proximal Policy Optimization (PPO).

Stage 1: supervised warm-up

The warm-up trains on question, Chain-of-Thought, and answer data. In the paper’s setup it lasts two epochs in most experiments, with up to five epochs for MathQA multiple-choice and up to ten for the numeric MathQA variant. This gives the policy a starting ability to produce a complete solution before reinforcement learning begins.

Stage 2: on-policy PPO

During the second stage, the policy receives a question and samples a reasoning path. The answer is extracted and compared with the ground truth. In the paper’s numeric tasks, a correct terminal answer receives reward 1, an extractable but incorrect numeric answer can receive 0.1, and an invalid or missing answer receives 0. Non-terminal actions receive 0. The total reward also includes a KL penalty against the warm-up policy. The reported PPO run uses 300 epochs, with KL coefficient 0.01 for program-based CoT and 0.05 for natural-language CoT.

The paper distinguishes two reasoning formats:

• N-CoT: a natural-language reasoning trace.
• P-CoT: a Python program whose execution produces the answer.

The training examples use the same questions as SFT. ReFT’s additional supervision comes from sampled paths and their rewards, rather than from extra annotated questions.

What the experiments show

The paper evaluates Galactica-6.7B and CodeLLAMA-7B on GSM8K, SVAMP, and MathQA. In its single-sample value-accuracy table, CodeLLAMA with P-CoT improves from 63.68% to 75.28% on GSM8K after ReFT, an 11.60-point difference. With N-CoT, the same model improves from 43.59% to 53.30%. The paper describes the former as close to a 12-point improvement. These figures are for the named models, datasets, and reasoning formats.

For smaller tested models, the paper reports P-CoT results for Galactica-125M, Codeparrot-small, and Codegen-350M. ReFT remains above the corresponding SFT row on GSM8K, SVAMP, and MathQAMCQ in that table, but those results do not establish a size-independent guarantee.

At inference time, the authors also sample 100 solutions and compare majority voting with reward-model reranking. ReFT plus voting improves over SFT plus voting by 8.6 points on average across the reported GSM8K settings. That is an inference-time combination, separate from the single-sample policy accuracy.

A limitation in the reward

The paper reports a reward-hacking failure on MathQA multiple-choice data. A sampled solution can contain an incorrect calculation but end with the correct option letter, causing an answer-only grader to assign reward 1. The authors use a longer warm-up for that setting and report numeric MathQA separately. This example shows why a terminal answer reward can be misleading when intermediate reasoning is not checked.

ReFT is therefore a particular SFT-to-PPO recipe with the paper’s reward design and training schedule. It is useful evidence that on-policy exploration can improve the tested math policies, while also showing that the reward and evaluation format determine what the policy learns.