Speculative Decoding, Beyond Text

01The mental model: propose, verify, commit

Any autoregressive generator can be written as

c is everything already known: a text prompt, image embeddings, encoded speech, speaker identity, a protein motif, or a robot’s current observation. zₜ need not be a word token. It can be a timestamp, acoustic-codec ID, image code, motion primitive, or quantized action.

Normal decoding asks the large target model for one next-token distribution at a time. Each result is needed before the next step can begin, so the target is called serially. Speculative decoding uses a cheap distribution q to propose γ future tokens, then asks target p to score those positions in one causal block. Causality is preserved: position i can see only the committed prefix and earlier proposals.

02Two exact algorithms: greedy equality and sampling correction

Exact greedy decoding

The draft proposes y₁…yγ. The target supplies its argmax for each proposal position plus one extra position. Accept draft tokens left-to-right while yᵢ = argmax pᵢ. At the first mismatch, discard that token and everything after it, then emit the target argmax at that position. If every proposal matches, accept all and emit the target’s extra “bonus” token.

This reproduces ordinary greedy target decoding, subject to the same logit processors, constraints, position semantics, and deterministic numerical path. A different kernel shape can cause a low-precision argmax tie to flip, so “same algorithm” is not automatically bit-identical on every backend.

Exact stochastic decoding

Let qᵢ and pᵢ be the draft and target distributions after temperature, vocabulary masking, top-k/top-p, repetition penalties, grammar constraints, and every other processor. Draw proposal yᵢ ~ qᵢ, then accept it with

If a proposal is rejected, draw the correction from

If all proposals are accepted, sample the bonus token from the target’s next distribution. This rejection correction is why the final sequence has the target model’s distribution even though candidates came from q. Leviathan et al. formalized this exact procedure and reported 2–3× improvements on their tested Transformer workloads—not a universal promise.

Averaged over proposals y ~ q, the acceptance probability at one position is Σₓ min(p(x), q(x)) = 1 − TV(p,q). Acceptance therefore measures distributional overlap, not simply whether the draft is “smart.” This is why draft-to-target alignment is the useful training objective.

03One round, slowly

1. Checkpoint: both caches describe exactly the committed prefix of length L.
2. Draft: draw or greedily choose γ tokens. Save each qᵢ value needed for acceptance.
3. Verify: run the target on the proposal block with a causal mask. Align logits carefully: the distribution for proposal yᵢ is conditioned on the prefix plus y<i.
4. Accept: scan left-to-right. Rejection ends the round; later proposals are invalid because their conditioning included a rejected token.
5. Emit: commit accepted tokens plus exactly one target correction, or all proposals plus one target bonus.
6. Reconcile: truncate/advance both caches and every auxiliary state to the new committed length.

A pure greedy verifier

copyfrom dataclasses import dataclass
from typing import TypeAlias

Token: TypeAlias = int


@dataclass(frozen=True)
class GreedyRound:
    """The immutable decision from one target verification block."""

    accepted: tuple[Token, ...]
    emitted: tuple[Token, ...]
    rejected_at: int | None


def verify_greedy(
    draft: tuple[Token, ...],
    target_argmax: tuple[Token, ...],
) -> GreedyRound:
    """Accept the matching prefix and emit one target-owned token."""
    if len(target_argmax) != len(draft) + 1:
        raise ValueError("target must score every draft position plus one bonus")

    for index, token in enumerate(draft):
        if token != target_argmax[index]:
            accepted = draft[:index]
            correction = target_argmax[index]
            return GreedyRound(accepted, accepted + (correction,), index)

    return GreedyRound(draft, draft + (target_argmax[-1],), None)

The function contains no cache operations. That separation is intentional: acceptance math should be testable with tiny scripted distributions before being entangled with model state.

04The draft-model contract

A draft need not be an independently trained smaller Transformer. It is any cheap mechanism that can propose the next units and, for exact sampling, expose their proposal probabilities. Separate correctness requirements from performance preferences.

Hard requirements for the simple exact algorithm

A subtle but liberating fact: the draft is allowed to be inaccurate or to ignore part of the conditioning; exact verification will correct it. That hurts acceptance, not correctness. The target distribution and the accounting around the actual proposal process are what must be exact.

Performance requirements

• Cheap enough: judge measured draft latency, memory traffic, synchronization, and transfers—not parameter count alone.
• Aligned enough: acceptance on the real domain, languages, modalities, temperatures, and prompt lengths matters more than perplexity in isolation.
• Conditioned enough: an audio-blind ASR or vision-blind VLM draft can remain lossless under exact verification, but usually wastes grounded proposals.
• Small working set: a draft that evicts target weights or KV from useful memory may make the target verification slower.
• Operationally simple: shared tokenizer, processors, device, and cache format reduce bookkeeping and bugs.

05Six ways to obtain proposals

Medusa is the canonical multi-head example; LayerSkip demonstrates early-exit self-speculation. DistillSpec shows why training the draft to match the target’s actual visited distribution can improve acceptance. These are not mutually exclusive: a server can select prompt lookup for copied spans, a small draft elsewhere, and disable speculation when recent acceptance collapses.

06Predicting speed before benchmarking

Let γ be proposal length and α the probability each next proposal survives, treated as constant and independent only for a rough model. The expected number of committed tokens per round is

If one ordinary target step costs Cₜ, each draft step costs C_d, block verification costs Cᵥ(γ), and bookkeeping costs C_b, a first estimate is

This is deliberately a latency model, not a FLOP model. Verification processes more tokens than a one-token decode step, but in one launch-rich block it may use the GPU far more efficiently. Conversely, long visual prefixes, large KV reads, saturated batches, or an expensive draft can make it slower.

What the simple equation hides

• α changes with proposal position, temperature, domain, and recent context. Report it by offset, not only as one average.
• Cᵥ(γ) is not constant: attention reads grow, kernel shapes change, and longer blocks may cross efficient tile boundaries.
• Draft and target may contend for memory, streams, or devices. CPU drafting can add transfers and synchronization.
• EOS, short answers, and early rejection truncate rounds. Prefill and non-AR stages dilute decode-only gains through Amdahl’s law.

07KV caches and state: treat each round as a transaction

Most broken implementations get the acceptance formula right and the state machine wrong. At round start, record the committed length L. Draft and target may write speculative KV entries, but none are final until acceptance is known.

The logit-shift invariant

If target cache ends at the committed prefix and you already retained its next-token logits, those logits score y₁. Feeding y₁…yγ returns logits that score y₂…yγ and the bonus position. Implementations that compare proposal i with output row i without accounting for this shift are off by one.

State that must roll back or recompute

With a static cache, rejected values may remain physically stored only if logical length, masks, and position indices guarantee they can never be read. “We overwrote the pointer later” is not proof. Padded cache attention requires an explicit valid-position mask.

08LLMs: the cleanest starting point

Recommended first implementation

1. Greedy, batch one, same tokenizer, small sibling draft, fixed γ=3 or 4.
2. Share identical prompt tokens but keep target and draft caches separate.
3. Implement one target block-verification call and transactional cache truncation.
4. Require exact token equality against ordinary greedy generation over odd prompt/output lengths.
5. Only then add exact sampling, adaptive length, batching, trees, or different tokenizers.

Where each drafter works

Framework support is an implementation detail, not an algorithm guarantee. Current Hugging Face assisted generation supports assistant models, prompt lookup, self-speculation, and tokenizer-translation variants, but batching/cache constraints vary by release. vLLM exposes several speculative backends. Check the installed version and benchmark your serving scheduler rather than copying an old configuration.

09Vision-language models

For a VLM that autoregressively emits text, the verification math is unchanged. The difficult part is conditioning. The target distribution depends on visual features, layout/position encodings, and special image tokens. A text-only draft may predict fluent glue words well and fail exactly where visual grounding matters.

Three architectures

VLM checklist

• Freeze and reuse the visual prefix within a round; match MRoPE/position IDs and image-token placement.
• Do not claim that speculative decode accelerates the image encoder or multimodal prefill.
• Report acceptance separately for visually grounded tokens and ordinary syntax if possible.
• Long visual KV prefixes can make every block verification expensive; model visual-token compression as a separate lever.
• Evaluate text/token parity for exact decoding and task quality (OCR, VQA, grounding) for approximate schemes.

Early multimodal studies report that speculative decoding can help VLMs, but performance is workload- and architecture-sensitive. Treat newer multimodal speculators and benchmarks as emerging evidence, not as a reason to assume an LLM draft will transfer unchanged.

10LM-based ASR: audio is not optional conditioning

Encoder-decoder ASR such as Whisper-like systems has a non-autoregressive audio encoder followed by an autoregressive text decoder. Encode the current audio once. For useful acceptance, both draft and target decoders should be conditioned on semantically equivalent audio state; an audio-blind draft can still be corrected exactly, but will tend to fail on acoustically determined tokens. Their projected cross-attention KV can remain separate if widths differ.

11TTS, audio, images, video, motion, proteins, and actions

The acceptance protocol is modality-agnostic; the factorization is not. First write down exactly what one autoregressive step means.

Multi-codebook audio needs an axis decision

A codec model may generate time-major frames, codebook-major streams, or one semantic stream followed by parallel residual codebooks. You may speculate across time only where the target is autoregressive across time; you may speculate within a frame only if that is the target’s declared factorization. Flattening IDs in a convenient order changes the conditional model. For a lossless implementation, validate exact codec-ID parity; waveform similarity and listening tests are necessary only after choosing an approximate acceptance rule.

Discrete and continuous outputs are different mathematics

The ratio/residual equations above are easiest for finite discrete vocabularies. Continuous AR models require density-aware acceptance and a way to sample the positive residual density; nearest-neighbor tolerance is an approximate method unless proven otherwise. Recent continuous-image speculation work develops specialized machinery—do not paste the categorical formula onto vectors.

The external-feedback boundary

12Serving, batching, and when speculation loses

Speculative decoding was first attractive at batch one, where one-token target calls underuse hardware and weight/KV traffic dominates. In a busy continuous-batching server, ordinary decoding may already fill the GPU. Variable accepted lengths also create ragged work and scheduling pressure.

Disable or reconsider it when

• prefill, image/audio encoding, codec decoding, or network queueing dominates end-to-end latency;
• the target is already small or the batch already saturates compute;
• outputs are so short that setup and extra model loading dominate;
• acceptance collapses under high temperature, domain/language shift, or grounded multimodal tokens;
• the draft consumes enough memory bandwidth or VRAM to slow/evict the target;
• verification uses a poor kernel path for its block shape, or rejected KV is replayed serially;
• the serving engine cannot pack variable-length proposal blocks efficiently.

Useful runtime policies

13A safe implementation path for decode-lab

Build correctness in layers. Keep ordinary eager generation as the oracle and switchable fallback throughout.

1. Pure verifier: unit-test greedy prefix acceptance and stochastic residual math with tiny hand-computed distributions.
2. No-cache reference: score the full committed prefix plus proposals and prove logit alignment.
3. Target block path: add causal multi-token verification; compare every relevant logit with the no-cache reference.
4. Transactional caches: checkpoint lengths, retain accepted block KV, truncate rejected suffixes, then synchronize the draft.
5. Greedy parity: exact IDs over many prompts, odd lengths, early EOS, γ larger than remaining output, and forced constraints.
6. Exact sampling: add processed p/q, ratio tests, residual correction, and statistical validation.
7. Performance: profile draft, target verify, cache reconciliation, synchronization, and end-to-end latency independently.
8. Only then: adaptive γ, trees/heads, multiple devices, batching, VLM/ASR/TTS conditioning, or approximate acceptance.

Audit of the current teaching skeleton

src/qwen3_lm/inference.py::InferenceEngine.speculative_generate is currently a conceptual sketch, not a production-ready speculative path. Before benchmarking it, account for these exact failure modes:

• After prefill, drafting starts by feeding the prompt’s final token again instead of using the next-token logits returned by prefill, so the draft context begins with a duplicated token.
• The target verification call scores merged without the target cache/prefix state, so it does not necessarily evaluate candidates under the committed context.
• Its output row i predicts the token after proposal i, yet the loop compares that row with proposal i; this is the logit-shift error from section 7.
• Accepted target tokens are then replayed through decode_step one at a time, paying the serial target work again.
• The draft cache has already advanced through every guess; advancing it again for accepted tokens, without rollback on rejection, desynchronizes its logical history.
• The routine implements a greedy comparison only; it does not implement exact speculative sampling.

14The benchmark and validation contract

A speedup without a matched decoding contract is not evidence. Freeze this table before running the GPU.

Report both decode-only and end-to-end speedup. Use warmups, synchronize device timing correctly, include the non-speculative target baseline in the same process, and publish regressions too. CPU structural tests can validate state and math; they do not prove CUDA kernel behavior, actual model quality, or GPU latency.

15One-page decision sheet

Primary sources and further study

1. Leviathan, Kalman & Matias (ICML 2023), Fast Inference from Transformers via Speculative Decoding — exact stochastic algorithm and original systems analysis.
2. Google Research, Looking back at speculative decoding — retrospective, intuition, and extensions beyond text.
3. Zhou et al. (ICLR 2024), DistillSpec — draft/target alignment through task- and divergence-aware distillation.
4. Cai et al. (ICML 2024), Medusa — multiple decoding heads and tree attention.
5. Elhoushi et al. (ACL 2024), LayerSkip — early-exit self-speculative decoding.
6. Sun et al., Block Verification Accelerates Speculative Decoding — verification need not be limited to a left-to-right rejection scan.
7. Gagrani et al. (CVPRW 2024), On Speculative Decoding for Multimodal LLMs.
8. SpecASR (2025 preprint) and Token Map Drafting for ASR (2025 preprint) — emerging audio-conditioned and model-free ASR approaches.
9. So et al. (ICCV 2025), Grouped Speculative Decoding for Autoregressive Image Generation.
10. Continuous Speculative Decoding for Autoregressive Image Generation — specialized treatment for continuous outputs.
11. VADUSA (2024 preprint) — speculative AR speech synthesis with an explicit quality/speed tolerance mechanism.
12. Hugging Face Transformers assisted decoding guide and vLLM speculative decoding docs — current framework interfaces; verify against the installed version.