Speculative decoding and self-speculative decoding

Introduction

Autoregressive decoding normally runs the target language model once for each generated token. Speculative decoding uses a cheaper draft model to propose a short continuation, then asks the target model to verify those candidates in parallel. Self-speculative methods use one model with a cheaper internal pass for the proposal and the full model for verification.

The speedup depends on the cost of the draft pass, the agreement between the draft and target distributions, the hardware, and the number of tokens accepted. It is a decoding method with measurable tradeoffs rather than a universal latency guarantee.

1. Standard autoregressive decoding

Given a prefix, a language model produces a distribution for the next token. The decoder samples or selects one token, appends it to the prefix, and repeats. Generating N tokens therefore requires N sequential target-model steps. The key dependency is causal: the distribution for position i depends on the token selected at position i − 1.

2. Draft and verify

Let q be the draft distribution and p be the target distribution for the same prefix.

1. The draft model generates K candidate tokens autoregressively. It feeds each sampled candidate back into the next draft step.
2. The target model receives the original prefix and the candidate continuation. With a causal mask, it produces the target distributions for the candidate positions in one batched forward pass.
3. The decoder checks the candidates from left to right and keeps the accepted prefix.
4. At the first rejection, the algorithm discards that candidate and the remaining draft suffix, keeping the earlier accepted tokens. It samples a replacement from the corrected residual distribution and continues decoding from that replacement.

The target pass also produces the distribution for one position after the candidate block. When every candidate is accepted, the decoder can sample that extra target token and make progress without another target call.

Exact acceptance

For a candidate token x sampled from q, the exact rule accepts it with probability

min(1, p(x) / q(x)).

If the candidate is rejected, the replacement is sampled from the normalized positive residual

max(0, p(x) − q(x)).

Under the assumptions of the algorithm and a correct implementation, this rejection-sampling procedure preserves the target model’s output distribution. The proposal can therefore reduce the number of expensive target steps without changing the distribution that the target model defines.

Approximate acceptance rules may use thresholds or other simplifications. They can be useful in practice, but their quality and distribution properties must be stated for the particular rule and evaluation.

3. Why the method can be faster

The draft model performs several cheap sequential steps, while the target model checks the candidate positions together. If the target accepts a long prefix, one target call produces several tokens. If the models disagree early, the draft work is partly wasted and the gain is smaller. Candidate length also has a cost: longer blocks add draft work and target-side verification positions.

The first speculative decoding paper reports about 2× to 3× acceleration for T5-XXL in its evaluated setting, with outputs matching standard decoding under its exact procedure. Those measurements are tied to that model, hardware, workload, and implementation. They do not establish the same multiplier for every model or service. See the reported evaluation.

4. Self-speculative decoding

Self-speculative decoding keeps one set of model weights but uses a cheaper approximation as the draft pass. The full model then verifies the proposed tokens. This can reduce memory and deployment duplication, while still paying for the full verification pass.

Several approximations can play different roles:

• Layer skipping omits selected layers during the draft pass and uses all layers for verification.
• Early exit stops the draft computation at an intermediate layer and uses the final layers during verification.
• Low-rank approximation replaces part of a computation with a lower-rank version for the draft pass.

These mechanisms should be named separately because they change different parts of the forward computation. For example, LayerSkip trains a shared model to support early exits and then uses an early-exit draft with full-model verification. The LayerSkip paper describes that training and decoding setup.

5. Sampling and evaluation

Temperature, top-k, and top-p change the distributions from which tokens are selected. An implementation must apply them consistently with its acceptance rule and record the configuration. A high draft acceptance rate alone is not enough to compare systems: report model pair or approximation, candidate length, hardware, workload, output length, and whether decoding is greedy, sampled, exact, or approximate.

A useful evaluation records both target-model calls and end-to-end latency. It should also check output quality under the same decoding settings. Exact speculative decoding has a distribution-preservation guarantee under its stated conditions; approximate variants need their own empirical quality check.