REFT: Reasoning with Reinforced Fine-Tuning

In the ever-evolving landscape of artificial intelligence, a breakthrough approach called Reinforced Fine-Tuning (ReFT) is changing how Large Language Models (LLMs) learn to solve mathematical problems. This innovative method, developed by researchers at ByteDance, addresses a fundamental limitation in current AI training approaches and opens new possibilities for enhanced mathematical reasoning.

The Traditional Challenge

Traditional methods of training AI models to solve math problems rely heavily on Supervised Fine-Tuning (SFT) with Chain-of-Thought (CoT) annotations. While effective to some degree, this approach has a significant limitation: for each math problem in the training data, there’s typically only one annotated solution path. This is akin to teaching a student by showing them just one way to solve each type of problem — limiting their ability to handle variations or develop flexible problem-solving skills.

Enter ReFT: A Two-Stage Solution

ReFT introduces a novel two-stage approach that fundamentally changes how models learn to solve mathematical problems:

Stage 1: The Warm-up Phase

The process begins with a conventional supervised fine-tuning period, typically lasting one to two epochs. This phase helps the model develop basic problem-solving capabilities and establishes a foundation for more advanced learning.

Stage 2: The Reinforcement Learning Phase

This is where ReFT truly innovates. Instead of continuing with supervised learning, the model enters an exploratory phase using Proximal Policy Optimization (PPO), a reinforcement learning algorithm. During this phase, the model actively generates multiple solution attempts for each problem and learns from both successes and failures.

How ReFT Discovers Multiple Solution Paths

The genius of ReFT lies in its ability to organically discover and learn from various valid solution approaches:

Dynamic Solution Generation

• The model continuously generates different attempts at solving each problem
• These attempts are created “on-policy,” meaning they come from the model’s current understanding
• The generation process is guided by learned probability distributions over possible solutions

Smart Reward Structure

ReFT employs a carefully designed reward system:

• Correct solutions receive a reward of 1
• Partially correct attempts (wrong answer but valid numeric response) get 0.1
• Invalid or incomplete solutions receive 0

This nuanced approach encourages exploration while maintaining focus on accuracy.

Controlled Exploration

To prevent the model from straying too far from reasonable solutions, ReFT implements a KL divergence penalty. This mathematical safeguard ensures that new solution approaches remain within a sensible range of the model’s initial training, balancing innovation with reliability.

Real-World Impact and Results

The effectiveness of ReFT has been demonstrated through extensive testing across multiple datasets:

Performance Improvements

• Up to 12-point accuracy improvements over traditional methods
• Consistent gains across different types of math problems
• Superior performance even with smaller model sizes

Versatility

ReFT shows impressive results with both:

• Natural language Chain-of-Thought reasoning
• Program-based solution approaches

Conclusion

ReFT represents a significant advancement in how we train AI models for mathematical reasoning. Moving beyond the limitations of single-solution training to embrace a more dynamic, exploration-based approach achieves impressive improvements in problem-solving capabilities. This breakthrough enhances current AI mathematical abilities and points toward a future where AI systems can develop more human-like reasoning flexibility and adaptability.