The inference-efficiency playbook — post-training and serving for faster models

01The mental model everything hangs on

Every technique attacks one of four multiplicative terms:

latency/cost ≈ (tokens generated) × (bytes moved per token) × (time per byte) ÷ (parallelism amortized over batch)

• Decode is memory-bound. One token at a time; every step re-reads all weights plus the whole KV cache. Arithmetic intensity is ~O(1) per matmul. Anything that shrinks bytes read per token (quantization, sparsity, KV compression, fewer KV heads) is nearly free speedup at small batch.
• Prefill is compute-bound. O(n²) attention, full matmuls. Attacked with chunked prefill, sparse attention, prefix caching — a different toolbox.
• Batch changes the regime. At large batch, decode approaches compute-bound: quantization's win shrinks, KV capacity and scheduling dominate. A trick that wins at batch 1 can lose at 64.
• Token count is a hidden dimension. Generating 2000 reasoning tokens to say "yes" is 2000× the decode cost. Post-training can shrink this directly — most people forget it.

02Post-training routes that buy inference speed

Changing the weights by training so the deployed artifact is cheaper. Six families:

2.1 Distillation into a smaller student

• Classic soft-target KD (Hinton, 1503.02531) → sequence-level KD (Kim & Rush, 1603.09785) → on-policy distillation that fixes the train/inference mismatch (GKD, 2307.13386; DistiLLM, 2401.10491) and reverse-KD so students stop covering modes they can't (MiniLLM, 2305.08445).
• Prune-then-adapt: Sheared-LLaMA (2310.06694) — structured 3D pruning + continued training beats from-scratch small models at 25% of the compute; Minitron (2407.14679) — prune width/depth and MoE experts, then a short KD re-alignment (~3% of original training cost).
• The canonical inference-aware ASR example: distil-whisper (2311.00430) — decoder-only distillation of Whisper (6→2 decoder layers, vocab cut, synthetic alignment retraining): 4.9× faster at <1% WER loss, plus a cascaded text-only correction model.
• Fal's pragmatics (see §07): online KD, offline KD and plain PEFT landed at the same quality on their task — so ship PEFT, it's the cheapest.

2.2 Training the model to emit fewer tokens

• Concise-reasoning / overthinking control: token-budget penalties in RL (2403.05587), CCoT curriculum (2412.20993), L1 length-controlled DPO (2503.04697), Chain of Draft (2502.18600) — think in ≤5-word drafts, ~90% token cut at near-parity accuracy.
• Vocabulary work: MiniLLM-style vocab distillation shrinks the lm_head (a 151k-row embedding is real decode cost); tokenizer retraining cuts token counts 20–30% for CJK/multilingual.
• Adaptive computation: CALM early-exit adapters (2205.14135) and Meta's LayerSkip (2404.16710) — train draft-at-bottom/verify-at-top inside one model; self-speculation by construction.

2.3 Training the decoding algorithm itself (MTP & draft heads)

The most active 2024–2026 front — detailed in §03. Post-hoc head-training jobs cost GPU-hours, not runs: Medusa (2401.10774), EAGLE-3 (2503.01840), DFlash (2602.06036), DSpark; DeepSeek-V3 (2412.19464) proved MTP heads trained in the main run serve as a free draft module; post-hoc MTP self-distillation (2602.06019) retrofits the trick onto existing checkpoints (>3× decode, <5% accuracy loss).

2.4 Quantization-aware post-training

You cannot PTQ your way below 4 bits cleanly — see §04 for the ladder: learned rounding (LSQ 1810.01875, GMLQ 2102.03586), EfficientQAT (2407.11062: ~25% of usual QAT cost), progressive UPQ FP16→INT4→INT2 (2506.09104), end-loss-guided PTQ GuidedQuant (ICML 2025), and Quantization-Aware Distillation (2601.20088): KL from the bf16 teacher into an NVFP4 student — the current default at the frontier when you can't rerun the original SFT/RL recipe.

2.5 Architecture surgery + fine-tune recovery

• MHA→GQA uptraining (2305.13245): shrink KV heads with a small fine-tune; KV cache shrinks proportionally.
• Sliding-window + global interleaves (Gemma recipe) → bounded KV for streaming.
• MLA low-rank KV latents (DeepSeek-V2, 2405.04434): ~93% KV reduction — wants training in the loop.
• Layer/block pruning with cheap recovery: LLM-Pruner (2305.15286), ShortGPT (2310.11102 — delete 25% of layers in minutes), SliceGPT (2401.15024 — orthogonalize then slice, no retrain).
• MoE compression: expert pruning/merging (I-MoE 2405.04534, Minitron expert slimming).

2.6 Inference-aware post-training: optimize the procedure, not the likelihood

• InfAlign (2412.19792): align against the actual test-time policy (best-of-N), not single-sample quality.
• Diffusion step-distillation is the same philosophy with a sampler instead of a decoder: consistency models (2303.01469), progressive distillation (2202.00512), DMD/DMD2 (2311.18828 / 2405.14867), adversarial ADD → SDXL-Turbo (2311.17042), MeanFlow (2502.06085). Deployed artifact runs 4 steps, not 50 — and the objective was chosen because of the sampler. See §06 for the 2026 production version.
• Model merging (DARE-TIES, soups) as cheap capability recovery after an efficiency fine-tune ate generality.

2.7 The full catalog — every post-training method that buys inference speed

Grouped by which term of the §01 cost equation it attacks. A method is listed only if it needs gradient steps on real data — pure training-free compression lives in the §04 toolbox. Every entry links to its representative paper.

03Speculative decoding: the 2026 frontier

Exact greedy/sampling acceptance (Leviathan 2211.17192, Chen 2302.01318) preserves the target distribution; the engineering race is all about acceptance length and drafting overhead:

• DFlash's argument: EAGLE is autoregressive — drafting 8 tokens costs 8 draft passes and errors accumulate, capping ~2×. DFlash instead uses a deeper draft model with bidirectional attention that unmasks a whole block of γ∈[8,16] tokens in one pass. The single pass is 2–4× slower than EAGLE's — but it replaces EAGLE's entire draft phase and drafts more, so it wins (~3× on Qwen3-8B / B200).
• Training detail worth stealing (Baseten): condition on fused hidden states from 5–6 evenly spaced target layers; freeze embed + lm_head; random anchors; CE loss with exponential position decay exp(−(i−1)/γ) — earlier draft tokens matter more.
• Fine-tuned models need retrained drafters. Fal (§07): an off-the-shelf MTP/DFlash head trained on the base model under-performs on the fine-tuned target; warm-started retraining on task data fixed acceptance. And drafters for copy-heavy workloads (OCR, ASR) get another free option: prompt-lookup decoding (2311.03511) — zero-cost drafts from repeated spans.
• Self-speculative, no extra params: Draft & Verify layer-skipping (2309.08168), Kangaroo's double early exit (2404.18911), speculative streaming (reuse the target’s own recent hidden states as drafts), QuantSpec (2502.10424). Per-request speculation-length RL: RESDO. Hybrid draft-large/verify-small: Speculative Cascades (2507.20127).
• The catch that papers bury: speculation degrades at large batch — draft FLOPs eat what bandwidth savings bought, and host-side overhead kills the win. Baseten's fix: run draft + verify as one fused forward so overhead is ~zero (§08).

04The training-free toolbox (PTQ) and where it breaks

Rule of thumb from the 2025–26 literature: optimize for the exact deployment format first (PTQ + a fast kernel), and only escalate to QAT/QAD when crossing into sub-4-bit weights, FP4/FP8 activations, diffusion latents, or long-context KV where quality collapse is real.

05Non-training engineering: the systems layer across model families

Everything below changes how a fixed set of weights is executed — no gradient steps. First the LLM-serving backbone (the canonical stack), then the per-family layers: kernels/quant, diffusion, audio, VLM, and edge.

5.1 The LLM serving backbone (the order practitioners try things)

1. Profile correctly first. Roofline per op. The traps that generate fake numbers: JIT warmup on first call (librosa/numba ≈ 600 ms one-time), launch-overhead-dominated tiny kernels, thermal/power throttling on laptops, and batch-size dependence. Report min-of-runs, interleave A/B, sanity-check outputs — see the SGLang loop-repetition incident in §08.
2. Kernels & graphs: fused attention — FlashAttention-2 (2307.09284), FlashAttention-3 (2407.08608), Triton, CUTLASS epilogue fusion (§06), CUDA Graphs to kill launch overhead, torch.compile, dequant-fused GEMMs (§5.2).
3. Memory: PagedAttention (2309.06180, 2–4× throughput), KV virtualization, host/SSD KV offload (Mooncake 2407.00079), multi-tenant LoRA serving (S-LoRA; Baseten: 10k fine-tunes on one GPU).
4. Scheduling: continuous/in-flight batching (Orca, OSDI'22), chunked prefill (Sarathi 2308.16369 / Sarathi-Serve 2403.02310), prefix caching via radix trees (SGLang 2312.07104), KV-cache-aware routing, prefill/decode disaggregation (DistServe 2401.09670) once the phases want different machines.
5. Parallelism judgment calls: TP wins latency, EP wins throughput (Baseten on GPT-OSS 120B); DP-attention for long-context batch scaling; Ulysses/Ring sequence parallelism with communication–compute overlap (fal's Ulysses Unbound; DeepSeek-V3's DualPipe).
6. Training-free decode acceleration: prompt-lookup drafting (2311.03511) — free candidates from repeated spans, absurdly effective for copy-heavy OCR/ASR output; tree attention over multi-candidate drafts; self-speculative layer skipping (Draft & Verify 2309.08168).
7. Measure with SLOs, not peaks: TTFT (prefill), TPOT/ITL (decode), and goodput — throughput subject to a per-user speed floor. "16× more users at ≥300 tok/s" is the number that ships.

5.2 Kernels where compression lives or dies

5.3 Diffusion & generative media (training-free)

5.4 Audio — ASR & TTS

• The engine, not the model: faster-whisper (CTranslate2 rewrite + int8) gave ~4× over the original PyTorch Whisper at identical weights — the classic proof that re-implementing the runtime is a first-class technique. Whistle/kaminari-style CUDA-graph engines are the same move in modern dress.
• Streaming architecture is the latency spec: chunked encoders, VAD, first-sound latency budgets for TTS (fal's Chatterbox Turbo: sub-150 ms), streamed acoustic-token decode, KV reuse across turns.
• Codec frame rate is the cost multiplier for speech-LLMs: 75 Hz residual-VQ tokens vs 12.5 Hz (Mimi-style) token rates is a ~6× decode-bill difference — decided at training time, paid at serving time (§02).
• Front-end hygiene: decode+resample+mono in one pass (torchcodec vs librosa: 2×) — it's ~2% of wall; fix once, then forget it (dante lesson).

5.5 Vision-language / multimodal

• Prefill dominates VLM cost (hundreds–thousands of image tokens): shrink or skip them at serve time — FastV (2403.06764) prunes image tokens in deeper layers, no training; the tuned variant is PruMerge (§02 catalog).
• Encoder-output caching: don't re-run the ViT across turns; cache visual embeddings behind the same prefix-cache machinery as text.
• Resolution/tiling budgets and MRoPE-friendly windowing are scheduling knobs with direct prefill cost; bounded-KV sliding windows enable long streaming OCR (kaminari territory).

5.6 Edge & heterogeneous compute

• llama.cpp / GGUF k-quants + graph rewrites; exllama2 GEMV kernels; KTransformers (SOSP'25) CPU/GPU hybrid MoE — experts in system RAM, attention on the GPU; Apple unified-memory stacks (MLX); ternary kernels for BitNet.
• The edge is where the whole stack must compose: batch-1, bandwidth-starved → quantized weights + paged KV + self-speculation all hit simultaneously, and every unfused op costs more than it does on HBM-rich datacenter parts.

06Case study — fal: sub-second Ideogram V4, 6.3× with no visible quality loss

blog.fal.ai, Jul 2026. 2.75 s → 0.44 s at 1K, framed as cost = (# forward passes) × (cost per pass) with every factor attacked:

The "Fast" tier collapses the two guidance branches; "Instant" additionally collapses step count — 6.3× over the bf16 many-step base. Every technique attacks a different term of the same equation, and they stack.

07Case study — fal: ~1000 tok/s and 16× throughput on the Ideogram prompt expander

blog.fal.ai, Jul 2026. An LLM pipeline end-to-end: expand user prompts (~600 tokens) in <2 s so the prompt expander stops being the bottleneck next to the (already distilled) image model.

• Size ≠ serving speed: at low concurrency, 0.8B through 27B dense models were almost equally slow — memory bandwidth. A 35B-A3B MoE in FP8 was the sweet spot: prompt expansion needs world knowledge; the 4B variant produced worse images and broken JSON despite few-shots.
• Distillation pragmatics: distilled the 397B teacher's style/format; online KD, offline KD and PEFT all converged to the same model → they shipped PEFT. Froze MoE expert weights during fine-tuning: training them made the model "ambitious beyond its capability," breaking image coherence.
• The speculation ladder, with measured rungs: 328 tok/s baseline (SGLang+FlashInfer) → 500 with the stock MTP head (not fine-tuned, acceptance ~2.5) → 468 with off-the-shelf DFlash (wrong distribution) → warm-start retrain of DFlash on 250K task samples: cold start failed (data not diverse enough; loading pretrained draft weights and continuing training fixed it) → 700 tok/s, 3.9 acceptance → DSpark (DeepSeek's DFlash + Markov refinement heads): reimplemented the trainer in TorchSpec to dodge its ~38 TB-of-hidden-states-on-disk requirement, warm-started from their own DFlash checkpoint, patched vLLM and SGLang to serve it at all → 830 tok/s at 4.6 acceptance in production.
• Plumbing > parallelism: TP=4 broke the sharded Markov heads' collectives — replicating the draft-head weights instead pushed past 1000 tok/s — but production stayed TP=1 because TP=4 wasted GPU cost. Result: 2.6× at single-user, 16× total throughput at a fixed 300 tok/s/user floor on one B200.

08Case studies — Baseten: GLM-5, DFlash, video, honest benchmarks

• Fastest GLM-5 API (SOTA on Artificial Analysis: 186+ tok/s, lowest TTFT). Custom DeepSeek-Sparse-Attention kernels: the lightning indexer must scan the whole sequence, so at short context sparse attention is slower than dense — production fix: skip the indexer below K, fuse the indexer's high-precision projections (they can't run FP8) through the overlap path. Zero-overhead MTP: draft and target run as one fused forward — "an often-overlooked aspect of speculative decoding is minimizing the overhead." Plus NVFP4 on Blackwell, hand-profiled MoE dispatch kernels, KV-aware routing.
• DFlash implementation (~3× latency and throughput vs 2× EAGLE, Qwen3-8B on one B200; GSM8K 654 TPS mean). Best public explanation of block-diffusion drafting + the training recipe (§03).
• "Fastest video inference, ever" (Wan 2.2): 5-second video, 133 s → 3 s on one B200 — custom Video Sparse Attention + AdaLN/RMSNorm/gated-residual fusion + timestep distillation + FP4. Convergent with fal's Ideogram stack, which is itself the finding.
• GPT-OSS 120B at 500+ tok/s: the TP-vs-EP judgment call — TP for latency SLOs, EP for raw throughput; shipped TP4EP1 on TensorRT-LLM's Blackwell MoE backend.
• Benchmark hygiene: their DFlash eval excluded SGLang because it produced loop-repetition outputs that inflated acceptance stats and biased length distributions — filtered results "were not directly comparable." The stack's numbers lie unless you eyeball the generations.
• Research track: "Still" (single-pass amortized KV-cache compaction), RadixMLP (intra-batch deduplication — the same tokens computed once per batch step, Aug 2026), neural KV-cache compaction for long-running agents, "BYO SWE-grep" (RL-trained fast code-search sub-agents), production speculative-decoding on TRT-LLM, and 10,000 fine-tuned LoRA models served from a single GPU.

09Case study — vLLM: the open engine room, same recipes

blog.vllm.ai, Dec 2025–Aug 2026. The closed-shop stacks of §06–§08 have a public mirror — same techniques, reproducible numbers, and the best place to watch the field institutionalize:

• Speculative decoding became a toolchain, not a trick. DSpark adaptive verification sizes each request's draft-check budget from the draft's own confidence — one configuration holds the throughput/latency frontier from batch 1 to 256, the direct answer to §03's "speculation dies at big batch." P-EAGLE drafts multiple tokens in one pass; EAGLE-3.1 adds post-norm hidden-state feedback; Speculators plus a hidden-state-extraction API make drafter retraining an offline job on task data — fal's §07 workflow, but pip-installable. AMD ships the same story: EAGLE-3 trained + served with Quark at 1.79–2.00× on Kimi-K2.5 / MiniMax-M2.5.
• KV cache is the platform layer. FP8 KV + attention is the shipped default (per-layer skip lists, NIAH before/after checks); the first TurboQuant study says 4-bit KV wins at long context but FP8 remains the safe pick; Decode Context Parallelism shards KV along the sequence across GPUs for ~3× long-context agentic throughput vs plain TP; Mooncake / PegaFlow external KV stores, an async KV-offloading connector, KV-aware routing, elastic expert parallelism and even Mamba conv-state transfer for hybrid SSMs (disaggregation for SSM-attention hybrids) make disaggregation plumbing, not research.
• The headline stacks converge on the §10 recipe: Qwen3.5-397B-A17B NVFP4 on GB200 NVL72 disaggregated → 25K total TPS/GPU; GLM-5.2-NVFP4 on 24×B300 → mean TPOT 40→17 ms via P/D disaggregation + MTP + Model Runner V2; DeepSeek at 2.2k tok/s/H200 on Wide-EP. Day-0 support now ships with model-native MTP, sparse attention (DSA, MiniMax sparse, Kimi-K3 KDA-aware prefix caching) and even the first diffusion LM (DiffusionGemma) reusing the speculative-decoding verification machinery.
• Media on the same engine: vLLM-Omni — TeaCache/Cache-DiT diffusion step caching (1.5–2× for free), staged TTS serving with CUDA graphs, and distributed layerwise offload running a 124 GB DiT on 64 GB HBM. The Qwen3-Omni serving postmortem lists its wins as hot-path cleanup and launch-overhead removal — §05.1 by name.
• Discipline, institutionalized: nightly performance + accuracy evals across every accelerator gate releases, and Inside vLLM: Anatomy of a High-Throughput LLM Inference System is the field's best single engineering document.

10Where it converges: the 2026 standard playbook

11Read-through for kaminari & dante

12The low-resource chapter: this playbook on a 6 GB laptop GPU

The case studies measured on B200/B300 fleets — but almost every §02/§04 lever is a fine-tune-scale job, and fine-tune scale fits in 6 GB with room left for discipline:

12.1 The experiments worth running first (open cells, not saturated ones)

• Vocab-diet KD on a 152k-vocab ASR decoder: the lm_head alone is ~155M params — a quarter of a 0.6B model. Distill into a 32k-row student on the deployment's text distribution; the output GEMM gets ~5× cheaper and the WER-vs-TPOT Pareto basically draws itself. distil-whisper proved vocab-cutting in principle (§02.1); nobody has published it for the Qwen3-ASR family.
• Task-domain drafters for copy-heavy decoders (ASR/OCR): drafters are trained for chat; transcription/OCR output is template-y — acceptance ≥5–6 is plausible where chat gets 2–4 (§03), and prompt-lookup gives a free baseline before training anything.
• Consumer 2:4 sparsity: the sparse-tensor-core literature benchmarks A100+; one-shot SparseGPT + cuSPARSELt on a 3050-class part is near-unexplored — and the sparsity×W4 interaction (do they stack?) is an open cell.
• QAD int4 recovery gated by a task metric (WER/NED, not perplexity): fal's §06 recipe transplanted from velocity-MSE to token logits is a genuinely open study.
• Self-gating loops: TTS→ASR round-trip WER as a human-free quality gate — the synthesizer generates the data, the transcriber scores it. Where two of your models validate each other, you can run sweeps with nobody in the loop (Baseten's "Still" KV compaction uses exactly this trick: ASR re-transcribes the compacted cache).

12.2 Protocol — cheap hardware punishes sloppy measurement twice

• Power/thermal caps are the enemy: a 35 W laptop part throttles within ~2 consecutive benchmark runs — fresh process, few runs, interleaved A/B, report min, checkpoint like preemption is imminent (it is).
• Disk is a first-class constraint (10–40 GB free is normal): audit the HF cache before downloading — and beware config-only stubs: a "cached" model can be 84 KB of metadata with no weights.
• Data without downloads: synthesize pairs locally (cached TTS generates audio, cached ASR + forced alignment generates transcripts).

12.3 When to rent: market rates, Aug 2026 (USD and NGN per GPU-hour, ₦1,338.68/USD)

The spread is the story: same-class H100 listings span $1.49–6.98/hr across providers, and marketplace prices move hourly (budget ₦1,400–1,500/USD for card fees and parallel-market reality). Working rule: iterate locally, rent only to generalize a finding — one overnight 4090 (~₦2k–10k) finishes a QAD or depth-prune sweep; the H100 tier is for headline benchmarks after the laptop says the result is real. Vast interruptible + checkpointing is just §12.2's thermal protocol with extra steps.

13The audio lane: the fal recipe on Stable Audio 3, and mini-DSpark for speech tokens

13.1 Running the §06 stack on stable-audio-3-small — what's left to attack

Stability already ran much of the playbook (§06's own lineage): the Small-Music/SFX post-trained checkpoints ship as a 433M-param flow-matching DiT over 256-dim SAME latents (4096× compression, 44.1 kHz stereo, 266M-param SAME-S autoencoder), pre-distilled and adversarially post-trained to 8 PingPong steps at CFG 1.0 — guidance is already folded and steps already collapsed from the base checkpoints' ~50 steps at CFG≈7. So the cost equation reduces to two live terms and one honest alternative:

• Cheap passes — quantize + QAD: a 433M DiT + 266M autoencoder quantize easily; int8 W8A8 with a short STE quantization-aware distillation (§06's recipe — the velocity field will not forgive quant noise any less than images) fits a 6 GB laptop. FP4 is Blackwell-only; on Ampere hardware int8 is §06's analogue.
• Fewer passes — 8 → 4 steps: TDM/CDM-style steps-aware distillation on top of an already-distilled model, quality-gated by FAD / embedding distance. Stability's own docs concede below-8 trades quality; the live question is whether a second distillation pass, warm-started into the student's own few-step distribution (§09's warm-start lesson applied to samplers), closes that gap.
• Reproduce the full stack from small-music-base: the pedagogically honest run — CFG-folding + DMD-family step-distillation + QAD, all on a 433M DiT. That's a fine-tune-scale job: ~₦33k–80k on rented 4090s for 3–7 days; pipeline smoke tests run on laptop VRAM.

13.2 Which small models can carry a mini-DSpark — and yes, someone has done it for speech

DSpark = DFlash's block-diffusion drafter + Markov refinement heads (§03). Its interface is architecture-agnostic: fused hidden states from the target, frozen embed + lm_head, standard speculative verification. What matters is whether the target's output token distribution is draftable. The local zoo:

Cost of a mini-DSpark: the drafter is ~one block + Markov heads reading the target's fused hidden states; 50–250K domain samples (fal needed 250K and warm-started, §07), and hidden-state dumps at 0.6B scale land in the tens of GB — nowhere near fal's 38 TB problem. Training is hours on a 4090 (₦3k–10k) or overnight on a laptop-class card, with the acceptance measurement + batch-scaling curve on the rented GPU. And there's the open cell, free for the taking: SpeechSpec published batch-1 numbers only — acceptance × batch × bit-width on ASR/OCR token streams (§01's regime shift meeting §12's laptop constraint) is uncharted.

14Source trail