kaminari — a single-GPU Qwen3-VL inference engine, explained

01TL;DR

kaminari is a from-scratch VLM inference engine: a native PyTorch reimplementation of the Qwen3-VL text decoder wrapped around the HF vision tower, a preallocated static KV cache, a torch.compile-shaped one-token decode loop, offline weight-only int4 quantization via torchao, and a FastAPI serving layer with three schedulers (microbatch, continuous slots, SSE streaming). It runs Qwen3-VL-2B at 100.6 tok/s decode — 3.49× the HF generate() baseline — in 2.3 GB of VRAM on an RTX 3050 (6 GB, mobile), and 9.35× aggregate HF throughput at batch 4.

The whole test suite (27 tests) runs on CPU against a ~1 MB synthetic checkpoint — no GPU needed to verify engine correctness. GPU-only paths are exercised by benchmark scripts on the real box.

02Why kaminari exists

The author runs OCR and local multimodal workloads (receipt transcription, page OCR, captioning, VQA) on a mobile RTX 3050 with 6 GB of VRAM. The stock path — HuggingFace Qwen3VLForConditionalGeneration.generate() — manages 28.8 tok/s on that card, spends memory on dynamic cache growth, and provides no control over batching or preprocessing.

kaminari's design constraints follow from that hardware reality:

• every MB counts — the model plus a 1024-token KV cache must leave headroom for the 6 GB card's other tenants; int4 weight-only quantization plus a preallocated cache cut peak VRAM from 4.2 GB to 2.3 GB
• compile shapes must be stable — variable image sizes churn torch.compile guards; fixed visual-token buckets pin the shapes
• OCR doesn't need sampling — greedy-only decoding removes the sampler from the hot path entirely
• one GPU owner — a single-GPU runtime doesn't need vLLM's distributed machinery; it needs a tight, auditable loop

The design borrows deliberately from three references: kestrel (moondream's edge inference stack — fixed-shape decode, prefix thinking), gpt-fast (pytorch-native hot path, static KV, weight-only quantization), and nano-vllm (minimal scheduler/paged-cache architecture). kaminari is what falls out of applying those ideas to a multimodal model on a small card.

03Architecture overview

src/kaminari/
├── __init__.py          # exports Engine
├── api.py               # FastAPI server, click CLI, pydantic schemas, SSE
├── vram_utils.py        # VRAM stat helpers
├── engine/
│   ├── types.py         # EngineConfig, Batch, Live, DecodeSlot(+Pool), StreamEvent
│   ├── frontend.py      # cpu preprocessing: bucket → chat template → processor → Batch
│   ├── processor.py     # HF processor loading, dtype + visual-bucket resolution
│   ├── runtime.py       # Engine: generate / generate_stream / prefill / decode / finalize
│   ├── scheduler.py     # MicrobatchScheduler, ContinuousBatchScheduler, StreamingScheduler
│   └── utils.py         # metrics, compiled decode wrapper, sampling, token commit
└── model/
    ├── qwen3vl.py       # Qwen3VLForConditionalGeneration (native) + load_qwen3vl
    ├── modules.py       # TextModel, DecoderLayer, TextAttention (fused QKV), MLP, Mrope
    ├── cache.py         # StaticKVLayer / StaticKVCache (preallocated, slot-addressable)
    └── utils.py         # rope/masks, checkpoint loading, projection fusion, int4 sidecar

Layer responsibilities

State model

Three immutable dataclasses carry the entire generation lifecycle, updated via dataclasses.replace() — state changes flow through return values, not mutation:

• Batch — one preprocessed request bundle: request ids, chat texts, padded tensor map from the processor, per-row prompt lengths, images, EOS/pad ids
• Live — hot decode state: static KV cache reference, accumulated generated_token_ids, next_token_id column, 3D M-RoPE position_ids, runtime cache_position, per-row finished flags
• DecodeSlot — a Live pinned to a physical cache row for the continuous scheduler; slots live in a fixed DecodeSlotPool sized max_batch_size

The one sanctioned exception to immutability: the KV cache buffers themselves mutate in place — that's the point of a static cache.

04Request lifecycle

blocking path (/generate)

Inside Engine.generate(), five stages run in order:

1 · preprocess (cpu)

Each image is letterboxed onto a fixed visual-token bucket canvas: the frontend picks the aspect-closest divisor grid for the token budget (e.g. 512 tokens → 16×32 → 896×448 px at 28 px/token for a wide image, 448×896 for a tall one), pastes the LANCZOS-resized image centered on a black canvas, then renders the chat template and calls the HF processor with padding="max_length" / truncation=True bounded by kv_cache_max_len − max_new_tokens. The result is one Batch whose padded tensors are trimmed to the batch max prompt length before anything moves to CUDA — pad tokens must never reach the cache.

2 · prefill (one padded batch forward)

Trimmed inputs move to device; the native get_rope_index computes 3D M-RoPE positions and rope_deltas on CPU (integer extraction stays off the GPU path); the static cache is built; a single forward runs vision encode + text prefill with logits_to_keep sliced to each row's last non-pad prompt token. The first token is sampled with greedy argmax immediately — TTFT is the prefill, there is no separate "first decode step".

3 · decode (compiled one-token loop)

Each step calls a torch.compile(mode="reduce-overhead", fullgraph=True)-wrapped closure with fixed-rank tensors only: token ids (batch, 1), positions (3, batch, 1), cache_position (1,), and the full preallocated KV buffers. Greedy selection argmaxes the final logits row outside the compiled callable and stays on-device until commit. Per-row EOS flips a finished flag; finished rows are padded with pad ids so the batch shape never changes mid-generation. The loop runs until every row is finished or max_new_tokens is hit.

4 · finalize

commit_tokens detaches GPU tensors → CPU → python lists, trimming each row at its first EOS; decode_text batch-decodes through the tokenizer. Metrics (preprocessing, prefill, decode, forward, sampling, finalize, TTFT, tok/s, img/s, VRAM) are assembled into EngineMetrics and attached to every row.

streaming path (/generate_stream)

The streaming endpoint bypasses batching entirely: a StreamingScheduler runs Engine.generate_stream() — a synchronous generator sharing the same prefill, then yielding one StreamEvent per decode step — in a worker thread, bridging events to the event loop through an asyncio.Queue. The API layer formats each event as data: {json}\n\n (SSE), checks request.is_disconnected() every cycle, and emits a final done event carrying the full text and metrics.

05The native model port

kaminari does not call HuggingFace's forward(). The text decoder is a native reimplementation in model/; the vision tower is reused from transformers. The split is deliberate: the decoder is where every decode step is spent and where compile/quantization control is needed; the vision tower runs once per request during prefill and HF's implementation is already fused and correct.

06The optimization ladder

Single-request decode throughput on the RTX 3050 (Qwen3-VL-2B, 512-token generations, 1024-token KV budget, greedy). Each row is a measurable milestone, kept in benchmarks_3vl/ with the exact run JSON:

what each step actually bought

• native port (+5%) — the win wasn't raw speed; it was control. Owning the forward unlocked the fused projections, the static cache, and compile. (The early HF-wrapper stage was honestly slower than stock — the table keeps that negative result.)
• int4 (+60→80%) — decode on a 3050 is memory-bandwidth-bound; halving weight bytes nearly doubles decode. VRAM dropped 45% in the same move.
• compile + static KV (+94% on top of int4) — the big one. Fixed shapes + reduce-overhead (CUDA-graph-backed) removed per-step kernel-launch and python overhead; static buffers removed cache-growth allocations. Together they took 5.18 s per 512-token generation.
• batching (+835% aggregate) — one padded prefill and one decode step serve N rows; aggregate throughput scales until the card goes VRAM-bound.

the concurrency curve

Concurrent /generate traffic with microbatching enabled surfaces the 3050's limits honestly. At 512 output tokens per request:

batch=2 is the sweet spot: batch=4 doubles latency and loses throughput because operations become VRAM-bound. The merged EngineConfig ships max_batch_size=2 as its default for exactly this reason.

07int4 sidecar quantization

kaminari treats quantization as an offline packaging step, not a runtime option: scripts/quantize_int4.py loads the bf16 model on a CUDA machine, converts only the language-model attention/MLP linears to torchao weight-only int4, and writes a sidecar pack — the non-quantized tensors (vision tower, embeddings, lm_head, norms) stay in a pruned model.safetensors, the packed decoder weights go to qwen3-vl-int4-pack.pt.

why this shape

• selective targets — only decoder linears are quantized; the vision encoder and lm_head stay bf16, so visual fidelity and the final logits projection keep full precision. The packer validates that no module outside model.language_model.layers.* attention/MLP projections was converted
• low-memory restore — the engine loads the pruned fp weights, moves to CUDA, then installs the int4 wrappers and reads the sidecar; torchao's tile-packed int4 requires the module to already live on the target device, a subtle ordering constraint the loader handles (and tests pin)
• explicit, not implicit — quantization="int4" fails hard unless int4_pack_path is provided; there is no silent fallback path that quantizes at load time

08The serving layer

three schedulers, one engine

• MicrobatchScheduler (default) — bounded asyncio.Queue (backpressure via max_queue_size), a short coalescing window after the first job so concurrent requests batch together, one Engine.generate() per batch in a worker thread, futures scattered back to rows. Per-job scheduler metrics (queue wait, microbatch wait, batch id/size) ride alongside engine metrics in every response.
• ContinuousBatchScheduler — admits requests into free DecodeSlots and ticks active slots one token at a time, finalizing each row the moment it finishes instead of waiting for the whole batch. The slot pool owns a shared static cache with per-row physical rows; copy_row_from migrates a finished row's KV state.
• StreamingScheduler — no batching, no slots: one stream at a time, token events bridged from the engine thread to the event loop.

The engine itself stays synchronous and batch-shaped — all async concerns live in the scheduler layer, and the API layer never touches tensors. Server defaults match the measured sweet spot: int4 on, max_batch_size=2, 50 ms coalescing window, kv_cache_max_len=2048.

09Correctness engineering

An inference engine that can only be tested on a GPU is an engine you can't refactor. kaminari's suite runs entirely on CPU: scripts/create_tiny_qwen3_vl_checkpoint.py synthesizes a ~1 MB Qwen3-VL-compatible checkpoint (tiny-but-structurally-exact config: 2 layers, 4 heads, the real processor and chat template) into tests/fixtures/tiny_qwen3_vl_cpu, and every test runs the real engine path against it.

27 tests, ruff check clean across src/ tests/ scripts/, deps pinned to CPU wheels via [tool.uv.sources] so uv sync + pytest works on any laptop. Testing locally is the whole story — no CI/CD, no remote runners.

10Repo state & roadmap

where it stands

• main carries everything: native engine, int4 sidecar, three schedulers, SSE streaming, green CPU suite, ruff-clean lint, MIT license, committed benchmark history (benchmarks_3vl/ JSONs back the README table)
• parked as tags — archive/qwen3-mtp (qwen3.5 adapter + MTP speculative decoding, 15 commits) and archive/decode-slots-v1 (slot exploration, 12 commits); restore with git checkout <tag>
• documented — docs/project.md maps every module, the request path, the scripts inventory, and the simplification proposal; logs.md carries the full edit history

next unlocks, in the order that makes sense

11Swap analysis: replacing Qwen3-VL with GLM-OCR

GLM-OCR (zai-org, Jan 2026) is a purpose-built document-OCR VLM — and a genuinely interesting swap candidate for kaminari, because kaminari's stated hot path (page/receipt OCR) is exactly what it was trained for. This section is a fact-first look at what the model is, how it compares, and what an engine port would actually involve.

11.1 · what it is

The structural overlap is the headline: same bridge shape, same patch-14 × merge-2 math (28 px/token — kaminari's bucket canvases transfer directly), same GQA-plus-M-RoPE decoder. GLM-OCR drops deepstack (single-level prefix features) and adds an MTP draft head kaminari's core doesn't currently use.

11.2 · the benchmark record

11.3 · what a port would take

Mapped onto kaminari's actual modules — the port is mostly mechanical because the bridge shape matches:

Roughly a focused week for a bf16 model-only port with green tests — call it the cleanest second-model port kaminari could ask for — plus open-ended work if the MTP path or the two-stage layout pipeline gets pulled in.

11.4 · verdict

Keep Qwen3-VL as the engine's general backend; treat GLM-OCR as the specialized OCR track, and let data decide the swap. Concretely:

• today, for OCR output quality: try GLM-OCR through vLLM ≥ 0.19 with its built-in MTP speculative decoding (--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}') or the official SDK for full-page parsing — kaminari's parked May experiment (scripts/vllm_image_benchmark.py, narrowed to zai-org/glm-ocr with an MTP speculative config, per logs.md) already sketched exactly this path
• for the engine: GLM-OCR is the best-justified second model/ package — the bridge matches, the cache/attention/M-RoPE machinery is reusable, and the 0.5B decoder flatters every optimization kaminari already ships — but only after the head-to-head
• the deciding experiment: run OmniDocBench-lite (or a fixed page set) through kaminari's scripts/ocr_eval.py with Qwen3-VL-2B bf16/int4 vs GLM-OCR model-only, on the same 3050, same visual-token budget. Quality + tok/s + VRAM on one table; then the swap is an engineering decision, not a vibes decision

12Source trail