Reinforcement Learning Fine-Tuning by OpenAI

OpenAI just announced Reinforcement Fine-Tuning. Here is the demo:

Reinforcement Fine-Tuning — 12 Days of OpenAI: Day 2

But what is Reinforcement Fine-Tuning (RFT)?

RFT is a new model customization technique that allows users to fine-tune OpenAI’s models (specifically the O1 series) using reinforcement learning rather than traditional supervised fine-tuning. The key difference is that while supervised fine-tuning teaches models to mimic inputs, RFT teaches models to develop new reasoning capabilities over custom domains.

How it works:

1. Data Preparation

• Users provide a dataset in JSONL format
• Each entry contains:
  • Input data (e.g., a problem or case)
  • Instructions for the model
  • The correct answer (used for grading, not shown to model during training)

2. Grading System

• Users define or use pre-built “graders”
• Graders score model outputs from 0 to 1
• Scores can be binary or partial credit
• The grading helps reinforce correct reasoning patterns

3. Training Process

• When the model sees a problem, it’s given space to think
• The model’s answer is graded
• Reinforcement learning algorithms:
  • Reinforce thinking patterns that led to correct answers
  • Disincentivize patterns that led to incorrect answers

Key Benefits

• Can achieve results with very small datasets (as few as dozens of examples)
• Allows models to learn new reasoning capabilities, not just mimicry
• Can make smaller models perform better than larger base models
• Works especially well for tasks requiring deep expertise

Example: Genetic Disease Diagnosis

The example in the demo is genetic disease diagnosis:

Input:

• Patient case reports containing:
  • Symptoms present
  • Symptoms absent
  • Patient history

Task:

• Identify genes potentially responsible for genetic diseases
• Rank genes by likelihood
• Provide reasoning for selections

Results shown in the demo:

• Base O1-mini: 17.7% accuracy
• Base O1: 25% accuracy
• RFT-trained O1-mini: 31% accuracy

The impressive part was that the fine-tuned smaller model (O1-mini) outperformed the larger base model (O1) after reinforcement fine-tuning.

Conclusion

This technique represents a significant advancement in model customization, allowing organizations to create highly specialized AI models for complex domain-specific tasks while requiring relatively small amounts of training data.

Apply for the RFT Research Program