Speculative Decoding and Self-Speculative Decoding: Faster Approaches to Large Language Model Generation

source

Introduction

The demand for real-time or near-real-time text generation from large language models (LLMs) has grown dramatically in recent years. Applications range from interactive chatbots and customer service agents to creative content generators, programming assistants, and more. As the size of these models continues to expand, so does the cost — in both time and computing resources — of generating text token by token. Speculative decoding arises as a compelling solution to this bottleneck. It cleverly uses two models (or two passes of the same model in some variants) to propose and validate multiple tokens in bulk, dramatically reducing inference time. In this post, we will explore how speculative decoding works, why it offers a speedup, and what trade-offs or implementation challenges might arise along the way.

1. Traditional Decoding: One Token at a Time

Before diving into speculative decoding, revisiting how standard autoregressive decoding works with a single large language model is helpful. Given a context of previously generated tokens (or an initial prompt), the model is used to predict the probability distribution for the next token. Then, one token is either chosen greedily (by taking the token with the highest probability) or sampled (to introduce randomness) from that distribution.

1. Context: We begin with a partial sequence, for example, “The cat sat on the.”
2. Forward Pass: We provide this sequence to the large language model, which outputs a probability distribution over its entire vocabulary for the next token.
3. Selection: We either pick the most likely token (greedy) or sample from the distribution. Suppose it picks “mat.”
4. Append: We append “mat” to the context, and the process repeats for the next token, “The cat sat on the mat …”
5. Repeat: For each new token, the model runs another forward pass.

In this scenario, if we need N new tokens, the model typically does N expensive forward passes, since each step depends on the output of the previous step.

2. The Core Idea Behind Speculative Decoding

Speculative decoding seeks to reduce the number of times the large model is called. Instead of relying on the large model to produce each token in a sequence (one pass per token), a smaller, faster “draft” model proposes multiple tokens. These proposed tokens are then validated by the larger “target” model in a more efficient manner, often requiring only a single forward pass for the entire chunk of tokens.

Broadly, the system looks like this:

1. Draft Model (Small): Generates a proposed set of K tokens in a single shot.
2. Target Model (Large): Performs one forward pass to validate the proposed K tokens.
3. Acceptance or Rollback: If most or all of these tokens are deemed probable enough by the target model, they are accepted in bulk. If not, the system reverts to the last accepted token and tries again, possibly with fewer tokens or a different approach.

By validating multiple tokens at once, we reduce the number of calls to the expensive large model. A key assumption is that the small draft model’s proposals align well enough with what the large model itself would have generated token by token, leading to minimal rollbacks.

source

3. How Speculative Decoding Works Step by Step

source

Let’s walk through a typical scenario using a sequence of tokens. Imagine we already have a context such as, “The cat sat on the,” and we want to generate more text.

1. Obtain the Proposed Tokens:

   The draft model (which is much smaller and thus faster to run) is provided the current context, “The cat sat on the.” We run it three times, and it outputs a proposed chunk of three tokens — for instance, [mat, and, purred].

2. Combine Context and Proposal:

   We then take the full context plus the proposed tokens: ["The", "cat", "sat", "on", "the", "mat", "and", "purred"].

3. Single Forward Pass on the Large Model:

   The larger, more accurate model sees the entire sequence in one pass. It outputs probability distributions for each position in that combined sequence. We specifically look at the distribution for the three new tokens — positions after “the.”

4. Check Alignment:

   For each newly added token, we compare the small model’s proposed token and probability with the large model’s probability distribution for that position. If “mat” has high probability under the large model (e.g., 0.35), we consider it well-aligned. If “and” also has a decent probability, it is good. If “purred” is drastically lower or not among the top probable words, the large model “rejects” that token. The system might roll back to the last good token, “and,” and then proceed from there by predicting the next token using the larger model.

5. Accept in Bulk or Roll Back:

   If the proposal is good for all tokens, we accept all three at once and skip calling the large model three separate times. Instead, we only used one forward pass to validate them, saving time and predicting the next token from all accepted tokens from the draft model. In the event of a partial mismatch, we only keep the tokens validated so far and might propose fewer tokens next.

4. Why This Speeds Up Generation

In standard decoding, producing K tokens means calling the large model K times, which is expensive. In speculative decoding, you might generate and validate those same K tokens with a smaller model and just one forward pass on the large model (plus a quick pass on the smaller model). Since smaller models can run significantly faster, the overhead of the draft model is minimal.

You thus replace K expensive passes with one expensive pass plus several cheap passes. If the acceptance rate of the proposed tokens is high, you realize a substantial speedup — frequently up to two times or more in many experiments.

5. Variants: Self-Speculative Decoding

An intriguing variant is self-speculative decoding, where there is only one set of weights. Instead of a separate smaller model, you use a cheaper approximation of the same large model (for instance, fewer layers or a low-rank approximation) as the “draft pass.” Then you do a full forward pass with all layers to validate the proposed tokens. This approach eliminates the need to maintain two distinct models, but still tries to capture the essence of generating multiple tokens cheaply and confirming them with the more expensive, complete version.

source

6. Sampling and Creativity in Speculative Decoding

Just like standard decoding, speculative decoding can employ sampling strategies such as temperature, top-k, or nucleus (top-p) sampling. You can apply these strategies either in the draft model’s token selection or in the acceptance threshold of the target model. This balances creativity and coherence:

• Temperature: Scales the probabilities of the next tokens, making them flatter or sharper.
• Top-k: Restricts sampling to the top k tokens.
• Nucleus (top-p): Samples from the smallest set of tokens whose cumulative probability exceeds p.

Each of these techniques can be layered onto speculative decoding for more varied text, while still reaping speed benefits.

7. Potential Pitfalls and Future Directions

1. Over-Reliance on the Draft Model: If the smaller draft model is not well-aligned, it may repeatedly suggest tokens that the large model rejects, reducing the speed gains.
2. Quality vs. Speed: Sometimes, a user might want absolute precision. Speculative decoding is generally safe but does introduce more complexity, and corner cases might lead to worse text if the fallback strategy is not robust.
3. Model Distillation: Methods to train smaller but highly faithful models can push speculative decoding further, minimizing rejections and maximizing throughput.
4. Self-Speculative Advancements: Researchers are experimenting with partial forward passes, low-rank approximations, or skipping layers to refine self-speculative techniques, aiming for large-scale improvements that do not require two sets of weights.

8. Conclusion

Speculative decoding represents a powerful strategy for accelerating text generation with large language models. By using a smaller, faster “draft” model (or partial approximation in the same model) to propose multiple tokens, and validating these tokens in a single pass of the larger model, we reduce the number of expensive decoding steps. This leads to considerably faster inference times, particularly in scenarios where latency is key — such as chatbots, user-facing applications, creative writing tools, or code generation assistants.

Yet, implementing speculative decoding does not come without challenges. Coordinating multiple models, deciding on chunk sizes, managing rollbacks, and ensuring minimal misalignment can be non-trivial in practice. Still, the documented speed gains — sometimes over twofold — make it a worthwhile addition to the modern AI toolkit. As both open-source and commercial frameworks continue to evolve, we can expect further innovations in speculative decoding, making next-generation language models even more practical and responsive.

References

• Fast Inference from Transformers via Speculative Decoding
• Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
• Looking Back at Speculative Decoding — Google Research
• Faster Text Generation with Self-Speculative Decoding — Hugging Face