Verifying LLM Outputs: The Evolution from Simple Consensus to Step-by-Step Analysis

As language models become increasingly powerful, ensuring their outputs are reliable and accurate has become crucial. Recent research reveals three progressively sophisticated approaches to verification, each building on the strengths of its predecessors.

Consensus Voting: The Power of Agreement

The simplest yet surprisingly effective approach is consensus voting. By generating multiple solutions and selecting the most common one, this method can significantly improve accuracy. Google’s Minerva AI demonstrated this dramatically — improving from 33.6% to 50.3% accuracy when using 1000 samples for consensus.

While effective for straightforward answers like numerical solutions, consensus voting shows limitations with complex outputs like mathematical proofs, where valid solutions might be worded differently but remain equally correct.

Best-of-N with Reward Models: Adding Intelligence to Selection

This method elevates verification by introducing a trained model to score solutions. Rather than just counting common answers, it evaluates each solution’s quality. The process is straightforward:

• Generate N different solutions
• Score each solution using a reward model
• Select the highest-scoring solution

However, this approach faces a critical challenge: its effectiveness depends heavily on reward model quality. Like a chain being only as strong as its weakest link, the method can suffer from overfitting if the reward model isn’t robust enough.

Process Reward Models: The Teacher’s Approach

The current state-of-the-art, introduced by OpenAI’s “Let’s Verify Step by Step,” represents a fundamental shift in verification strategy. Instead of evaluating just the final answer, it examines each step of the solution process — much like a teacher grading a student’s work.

The results are impressive:

• Consensus: ~70% success rate
• Best-of-N: 72.4% success rate
• Process Reward Models: 78.2% success rate

Consider a real example from the research: when tasked with simplifying “10 of 100 degrees + 4 times sine of 100 degrees,” raw GPT-4 succeeded only once in 1000 attempts. With step-by-step verification, the success rate improved dramatically.

Here are the results from the paper “Let’s verify step by step” by OpenAI which you can find here: 2305.20050

The Broader Implications

This evolution in verification methods reveals a fundamental truth about AI development: breaking down complex tasks into smaller, verifiable steps can lead to substantial improvements in reliability and performance. It’s a principle that mirrors human learning and evaluation methods, where understanding the process is often as important as reaching the correct answer.

The success of process reward models also points to a future where AI systems might not just solve problems but demonstrate their reasoning in a way that humans can verify and understand. This transparency could be crucial for applications in fields like education, scientific research, or any domain where the path to the answer matters as much as the answer itself.

As these methods continue to evolve, they’re likely to play an increasingly important role in making AI systems more reliable, transparent, and trustworthy — essential qualities as these systems become more integrated into our daily lives and critical decision-making processes.