Stable Audio 3 Small on an RTX 3050 6 GB — measured latency, per stage

Written 2026-09-14; rewritten 2026-09-15 around measurements. Target machine is victoria: NVIDIA GeForce RTX 3050 6 GB Laptop GPU (GA107, compute capability 8.6, 6144 MiB VRAM), 7.5 GB host RAM. It is the machine behind the Whistle runtime analysis and the Chatterbox V3 profiling pass.

Stable Audio 3 Small is a 433 M-parameter latent flow-matching transformer over a 4096× downsample autoencoder. It produces 120 s of 44.1 kHz stereo in 8 sampling steps. Measured here: 120 s of audio in 4.18 s cold and 2.16 s warm — 12.8× and 53× faster than real time respectively — at a flat 2179 MiB peak.

Everything below is tagged:

• measured — a number produced by running on victoria, with the harness named.
• derived — computed from measured or shipped values, formula shown inline.
• vendor-doc — read out of a released config, a line of source, or Stability's own benchmark table. Not re-measured here.

Nothing is presented as measured that was not. Where an earlier analytic pass and the measurements disagree, the measurements win, and two cases where the analytic model was structurally wrong are called out.

Measured: end to end

Harness hummingbird/scripts/generate.py. Model small-music, prompt a Ludwig Göransson-style orchestral cue, seed 20260915, steps=8, cfg_scale=1.0. Environment: stable-audio-3 installed editable from the upstream checkout, running on torch 2.13.0+cu130 / torchaudio 2.11.0+cu130, driver 580.119.02. The library pins torch==2.7.1; that pin could not be installed on this box because its nvidia-cuda-* dependency chain failed to download, so a pre-existing 2.13 build was reused.

Model load was a near-constant 9.6–11.4 s and is excluded from the generate column. Least-squares fit over the four points:

generate_seconds ≈ 1.92 + 0.01805 × duration_seconds

A fit on the 5/30/60 s points alone predicts 4.09 s for 120 s against 4.18 s measured — within 2 %, so the relationship is genuinely linear across the whole supported range.

Three things this establishes, in order of how much they change the picture:

A ~1.92 s fixed cost per generate() call that the roofline model omitted entirely. Duration buys almost nothing: latency is dominated by a constant term, not by per-latent work. At 5 s the fitted line already predicts 2.01 s, and 2.07 s was measured — the constant and the slope together account for both. Any latency model for this family needs a constant, and the previously published estimate of "0.6–1.3 s for 30 s" was not merely imprecise, it was structurally wrong because it attributed all cost to duration. The per-stage profile below identifies that constant: it is the text encoder's first call in a fresh process, and it is 100–200× cheaper on every subsequent call.

The analytic band was wrong in the optimistic direction, not the pessimistic one. It predicted 1.9–3.8 s for 120 s (RTF 0.016–0.032). The measurement is 4.18 s, i.e. 1.1–2.2× slower than the band; at 30 s it is 2.54 s against a predicted 0.6–1.3 s. The cause is the missing constant term: the roofline had no way to represent cost that does not scale with output length.

Peak VRAM is 2179 MiB and identical at every duration from 5 s to 120 s. This is a flat line, and it is the strongest available confirmation that resident cost is weights plus a fixed workspace while duration is nearly free. It also matches the derived 1.65 GB fp16 weight sum plus a ~0.5 GB context/workspace allowance. Only 35 % of the card is used, so "will it fit" was never close on this GPU, and the 4 GB variant would likely pass with room to spare.

Per-stage and per-module latency

Harness hummingbird/scripts/prof.py, which wraps the reference library's own modules from the outside (a CUDA event pair straddling each stage, resolved after torch.cuda.synchronize()) so nothing under refs/ is modified. Same prompt, seed, steps=8, cfg_scale=1.0 as above. Each configuration is run three times in one process: call 1 is the cold path a first request takes, calls 2–3 are the steady state.

conditioning is the T5Gemma text encoder plus the multi-conditioner; dit_forward is the diffusion transformer across all 8 sampling steps; decode is the SAME-S decoder; postprocess is the fp32 → clamp → int16 → host transfer.

Cold versus warm — this is the whole story

The ~1.92 s fixed cost is the text encoder's first call. conditioning costs 1.35–2.13 s cold and 10–21 ms warm — a factor of 100–200×. Every generate.py run in the table at the top of this article was a fresh process, so all four of its measurements paid that cost, and the fitted intercept of 1.92 s is almost exactly its mean. The constant was never an inherent property of the model; it was a cold-start artifact that the per-process harness could not see.

Subtracting it, the two regimes are both clean linear fits over the canonical durations:

cold:  generate = 2.042 + 0.01719 × duration     R² = 0.79
warm:  generate = 0.290 + 0.01631 × duration     R² = 0.97

The slope barely moves (17.2 → 16.3 ms per audio-second). All of the improvement is in the constant: 2.04 s → 0.29 s. The warm fit's residual structure is one outlier at 60 s; on the other three points it is within 0.08 s.

Reconstructing cold 120 s from its parts, 1.41 + 0.73 + 0.043 + 0.011 + 1.41 (warm DiT + decode + postproc + warm conditioning + cold conditioning premium) = 3.60 s against 3.83 s measured. The budget closes.

Steady-state RTF is 0.0187 at 120 s, not 0.0348 — 53× real time rather than 29×. The published figure is a cold-start figure, and any comparison against it should say which regime it is in.

The latency budget, ranked

Everything below is steady state at 120 s, one warm request, 2.171 s total, GPU clock already ramped and no other client on the card. Two views of the same measurement: what the stage costs, and what the hardware component underneath it costs.

By stage:

By component, inside the two stages that matter:

The single most important observation from these two tables: the decoder and the DiT run the same large GEMM kernel. The 160x128x32 tile shape is the largest single kernel in both stages — 343.0 ms in the DiT and 186.7 ms in the decoder, 529.6 ms and 24.9 % of all device time between them. The decoder is not a different kind of work; it is the same kind of work invoked 14 times in a row on small slices.

Three things that are not bottlenecks at 120 s, all measured:

And one that is a bottleneck only in a specific regime:

That last row is a 100–200× swing, and it is a host-side stall rather than slow arithmetic — see the cold encoder trace. A service that keeps a process warm never pays it; a CLI invocation pays it on every run, and at 5 s of audio it is over 60 % of the wall time.

Where the warm time goes

At 120 s, the warm 2.164 s decomposes as:

Two stages are 99 % of the request. The text encoder — which dominates the cold path — is free once warm, and post-processing is 43 ms for 120 s of stereo audio. The unaccounted term is negative by 1.7 %, which is the sampling loop's own host work (per-step schedule indexing, a torch.ones allocation, and the noise draw) falling inside the measured DiT window; the stages are not perfectly disjoint. The same decomposition closes at 30 s to within 0.4 %.

Both dominant stages scale linearly with duration, and the decoder scales slightly faster:

warm DiT   = 0.233 + 0.01028 × duration     R² = 0.97
decode     = 0.041 + 0.00599 × duration     R² = 0.98

Decode is 20 % of DiT at 5 s and 52 % of it at 120 s, so the autoencoder becomes a first-class cost at long durations — the opposite of the "decoder is a rounding error" instinct.

Requests above 120 s are silently clipped, not rejected. Asking for 240 s returns audio_seconds = 120.001, and a 240 s run's DiT and decode times are identical to the 120 s run's to within 2 % (1.428 vs 1.414 s, 0.736 vs 0.734 s) because no extra work happens. The 120 s ceiling in the model table is enforced by truncation, so a caller that requests a longer clip gets a plausible-looking shorter file rather than an error. That is worth knowing before timing anything above the cap.

Inside the DiT: 20 blocks, and they are uniform

Hooking each of the 20 transformer blocks across all 8 steps gives 160 timed invocations per call. Mean time per block, warm:

The blocks are flat: at 120 s every block lands between 8.62 and 8.72 ms (±0.6 %), and the same holds at 30 s (2.88–2.96 ms). There is no hot block and no pathological first layer to single out once warm. The block sum accounts for 97.6 % of the DiT total at 120 s, the remainder being the input/output projections and the timestep embedding.

Cold, this is not true. On call 1 the maximum block time is 17–22 ms against a mean of 1.5–8.7 ms — one block absorbing a one-time 17–22 ms cost, consistent with kernel autotuning on first execution. That is the DiT half of the warm-up penalty; the encoder is the other, much larger, half.

Per-step DiT latency from the trace

The block hooks give a mean; the Perfetto trace gives actual per-step timings, because record_function brackets each dit_forward call. At 120 s, warm, the eight steps are uniform:

Steps 1–8 agree to ±0.4 % and issue an identical 1,704 kernel slices each, which is the same conclusion the block hooks reach from a different direction: nothing inside the denoising loop varies by step. Device time per step (173.7 ms) is 99.4 % of its wall time, so the loop is not waiting on the host.

Measurement caveat. The trace contains dit_forward ranges on two worker threads (tids 7 and 19862), both running the full eight-step loop in overlapping intervals. Naively summing all dit_forward ranges therefore double-counts, and it is easy to mistake one thread's partially-observed first range for a fast first step — reading tid 19862 alone shows a 67.6 ms step 1 with 675 slices against 174 ms for its steps 2–8, which looks like a 2.6× warm-up advantage but is an artifact of where that range was measured. The table above is tid 7, whose first range is complete. The profiler runs a second thread here; whether that is a data-parallel path in the library or an artifact of Kineto's per-thread attribution was not established.

The step-count lever is small

At 30 s, steady state, steps=8 costs 0.715 s and steps=4 costs 0.474 s — a saving of 0.24 s, or 34 %, not the 50 % the step count suggests. The reason is that only the DiT shrinks: 0.491 → 0.264 s, while decode holds at 0.210 s and the remaining 0.01 s of conditioning and post-processing does not move. Decomposing the steady-state 30 s request, 0.491 + 0.210 + 0.006 + 0.011 = 0.718 s against 0.715 s measured.

So halving the step count buys a third of the time, and it costs an output-distribution change: because the sampler injects fresh noise at every step, steps=4 is a different render rather than a cheaper one, and the saving has to be bought with an audio A/B rather than a latency number alone.

Traces

The figures below are Perfetto renders of an actual warm 120 s request on victoria (and one cold 30 s request), captured with torch.profiler and NVTX-labelled per stage. They are the source of every per-kernel number in this article.

They come in two magnifications. The whole-run views show where time goes across a request; the deep zooms below each of them crop the trace to a sub-millisecond window and render only the GPU track, so individual kernel names and durations are legible. A 174 ms denoising step cannot be read at a glance any more than a 738 ms decode can — the zooms are the part that answers questions.

The whole request

Eight dit_forward passes run back to back, then a single same_decode runs at the end and takes a disproportionate share of the wall time.

One denoising step

Zoomed to three consecutive dit_forward ranges. The steps are byte-for-byte similar in shape and duration, which is why the per-block hook reports a flat mean.

1 ms inside that step

One millisecond of the step, GPU track only. The repeating unit is visible: a large GEMM, a flash_fwd attention kernel, then a dense band of 1–2 µs elementwise kernels.

The GEMM and attention kernels

500 µs at higher magnification. sm80_xmma_gemm_f16f16_f16f32 runs 1,085 µs on average across the whole run; here it sits next to the short kernels that surround it.

A single cutlass kernel

700 µs around a 104 µs cutlass_80_tensorop_f16_s16816gemm — the largest single DiT kernel in the step. The full kernel name is readable at this zoom.

The 1–2 µs tail

700 µs of the short-kernel tail. This band is why one DiT step issues 1,704 kernel slices while only a handful are large: the slice count is dominated by kernels that cost about a microsecond each.

Inside one step

Nine milliseconds inside a single dit_forward, deep enough to separate individual kernel slices. The 20 blocks produce a dense, regular band of short kernels.

The dominant GEMM

The longest sm80_xmma_gemm_f16f16_f16f32 slice in the trace, selected. This tile shape (160x128x32) is the single largest kernel by total device time at 24.9 %.

The decoder — the hotspot

The whole same_decode range at 120 s. It is 738.7 ms of wall time in one stretch, against 174.8 ms for a single DiT step — larger than any individual denoising step and 33.9 % of the warm request.

Inside the decoder

Fourteen milliseconds inside same_decode. The kernel mix here is visibly dominated by short elementwise and copy slices rather than the large GEMMs that dominate the DiT — the per-kernel signature of the 58.9 % elementwise share in the table below.

500 µs of the decoder

Half a millisecond, GPU track only. The band is almost entirely at::native:: elementwise work with a flash_fwd kernel in the middle — no large GEMM in this window at all.

At a chunk boundary

400 µs at the start of a decode window, where one of the 14 chunk passes begins. This is the repetition that makes the decoder 14 sequential invocations rather than one: the same sequence restarts every 96 latent frames.

The decoder's large GEMM

700 µs around a 555 µs cutlass_80_tensorop_f16_s16816gemm_relu_f16_128x256 kernel — the largest single kernel inside the decoder, and only the third-largest class share. Note the relu fused into the GEMM epilogue.

The elementwise share, up close

700 µs of decoder elementwise, copy and normalisation kernels. Individually they are 40–270 µs; collectively they are 432.7 ms, the single largest share of the decoder's device time.

The cold text encoder

The conditioning range on the first call in a fresh process. This is the 1.4–2.1 s cost that generate.py paid on every run and that the fitted 1.92 s constant corresponds to.

Cold, at 450 µs

450 µs inside the cold conditioning range, at the point where its compute actually happens. The encoder's kernels are here: ampere_bf16_s16816gemm_bf16 and fmha_cutlassF_bf16 attention. The text encoder runs bf16 even though the model loads fp16 — the trace is unambiguous about it.

The timing is the striking part, and it is a stall rather than slow arithmetic. Of the range's 1,494.9 ms, the GPU issues kernels in two thin slivers and one burst:

All 96 of the encoder's bf16 kernels land between +1,339 ms and +1,384 ms — about 45 ms of dense GPU work, arriving after more than a second in which the card is essentially unused. That is what the 1.4–2.1 s cold penalty is: not the encoder computing slowly, but the GPU waiting while the host prepares to use it. Warm calls skip the wait entirely, which is why the same kernel mix costs 11.9 ms.

Warm, at the same scale

The same 300 µs scale on the warm call, which reaches its bf16 kernels immediately: the warm encoder is 11.9 ms total against 1,425 ms cold. The kernel shapes are the same ampere_bf16_s16816gemm_bf16 work — the 120× difference is entirely in when the GPU is allowed to start, not in the steady-state kernel mix.

GPU stream occupancy

The GPU stream row across the last DiT steps and into the decode, showing launch density and the gaps between kernels.

Architecture in numbers

DiT constants (vendor-doc, from the MLX port's hard-coded sa3-sm-music block):

The forward pass sees T = L + 64 tokens, where L is the latent frame count and 64 is the memory-token prefix shared across the batch.

Latent geometry: 4096 samples per channel per latent frame, i.e. 44100 / 4096 = 10.7666 latent frames per second (92.88 ms per frame). Three independent confirmations agree: the TensorRT README bounds the DiT engine at "L=1..4096 = ~93 ms .. ~6.3 min" (4096²/44100 = 380.4 s ✓); the autoencoder benchmark states "L=8192 is 761 s" (8192 × 4096/44100 = 760.9 s ✓); and the SAME-S decoder profile ceiling L=1292 is exactly 1292 × 4096/44100 = 120.0 s, the small model's maximum ✓.

Correction. The model overview claims "for a 10-second clip, 2 channels × 441k samples compresses down to 216×256". That is wrong by 2×: 10 s is 108 latent frames. The 216 figure divides the interleaved element count (2 × 441,000) by 4096, while the per-channel divisor is also 4096. Every attention and activation figure scales with the token count, so this matters.

SAME-S is a separate, independently trained stereo autoencoder — not a diffusion VAE and not fine-tuned with the DiT. Vendor-doc from SAME-S/model_config.json: latent_dim 256, downsampling_ratio 4096, a patched pretransform with patch_size 256, and 6 transformer blocks per side (independently confirmed by the fp16 build script, which counts "24 instances total (6 layers × 4 norms/layer)" and "6 Cos + 6 Sin"). Two structural details drive the numerics later: its normalisation is not RMSNorm but a tanh-bounded affine, out = beta + gamma * tanh(alpha * x), and both sides use differential attention — two softmax maps subtracted, Sub(MatMul1(softmax1, v), MatMul3(softmax2, v)). SAME-S's scheme is documented as "a workaround for sliding window attention on CPU"; SAME-L uses real sliding-window attention and needs a GPU kernel for it.

There are no conv-only or vocoder stages. The decoder is the vocoder: audio is patched to 512 channels at 1/256 rate, encoded 16× down to 256-dim latents at 1/4096 rate, and the decoder inverts that.

T5Gemma is an embedding-bandwidth problem, not a compute problem. Derived from the shipped fp16 export: t5gemma_f16.npz is 567.4 MB, so ≈ 284 M parameters. Those are overwhelmingly a table: 768-wide with a ~256 k Gemma vocabulary gives ≈ 197 M embedding parameters (69 % of the encoder), the rest in ~12 layers. It runs exactly once per prompt.

FLOP budget: attention is not the bottleneck

Derived from the confirmed dims. MACs per token per block, excluding the text-side cross-attention K/V projection (computed on the short context, not on T):

Total per forward = 2 × 20 × T × (20,186,112 + 2048·T) FLOPs. At the small model's 120 s ceiling, attention is 12 % of the FLOPs and the linear GEMMs are 88 %; at 30 s attention is under 4 %.

Solving 2048·T = 20,186,112 puts the crossover at T ≈ 9857, i.e. L ≈ 9793 latent frames ≈ 909 s of audio — 7.6× beyond the small model's 120 s cap. Quadratic attention is not a bottleneck for this model at any duration it supports, and a 3050 owner should not spend effort on attention kernels. (Full self-attention with no KV cache is still why the model cannot be trivially chunked for memory; that is a different problem, and chunked decoding is the mechanism that addresses it.)

One calibration point from the vendor's own H200 table: the small model takes 0.45 s for 120 s of audio and 0.41 s for 5 s — flat in duration while the FLOP count rises 24×. On a large GPU the DiT's own FLOPs are almost free and everything is launch overhead. On a 3050 that reverses, and the constant term above is what replaces it.

VRAM: 2179 MiB, and why it is flat

Weights, all fp16 unless noted (derived from confirmed parameter counts):

Measured peak was 2179 MiB = 2.13 GB, consistent with those weights plus a ~0.5 GB context and cuDNN/cuBLAS workspace allowance. The measured figure is flat across a 24× range of durations, which is the direct evidence that activation memory does not grow with output length the way the token count does: sequence length is the state variable for compute, not for VRAM.

Measured memory bandwidth

The 144 GB/s figure that appears elsewhere in the SA3 write-ups is a datasheet product (96-bit bus × 12 Gbps), not a measurement. It was measured on victoria (scripts/bench_membw.py), and the datasheet is optimistic.

Achieved bandwidth is 128 GB/s — 89 % of the datasheet, 97.4 % of the clock-derived ceiling. The datasheet overstates this part by 12 %. Note the device's own memory_clock_rate is 5486 MHz, i.e. ~11 Gbps rather than the 12 Gbps the datasheet quotes, which is most of the gap.

Two other device facts worth correcting: L2 is 1,572,864 bytes (1.5 MiB), not the 2 MB assumed elsewhere, and there are 20 SMs (matching the DiT's 20 blocks, coincidentally). The L2 transition is measurable — bandwidth peaks at 158.6 GB/s for 256 KiB working sets, leaves L2 at 1 MiB, and plateaus at 122 GB/s from 8 MiB upward.

Consequences for the weight-streaming floor, which is why this was measured:

Even at the measured rate, re-reading every weight once per step is 54 ms across 8 steps against a measured warm DiT time of 1,414 ms — so weight streaming is 3.8 % of the DiT stage, and the model is compute-bound on this GPU by a factor of ~26. The correction matters for the cost model but does not change that conclusion.

Sampling: 8 steps, stochastic, and duration-dependent

Default steps=8, cfg_scale=1.0, sampler pingpong.

The schedule is warped in log-SNR, and the warp depends on the token count. logsnr ramps linearly from logsnr_end=2.0 down to an adaptive start of -6.2 - log₂(T/2000), then t = sigmoid(-logsnr), with the sigma_max=1.0 endpoint restored. Measured by calling build_schedule from the reference library directly:

The same 8 steps spread very differently across the trajectory at different durations — a longer clip spends its early steps much closer to the noise end. The ping-pong update is stochastic, not deterministic Euler:

denoised = x - t_curr * model(x, t_curr)
x = (1 - t_next) * denoised + t_next * torch.randn_like(x)

Fresh noise enters at every step, so seed reproducibility depends on the RNG stream and the step count. The library also implements discrete_euler, rk4 and flow_dpmpp; pingpong is the distilled-model default.

Classifier-free guidance is off by default and inert on post-trained checkpoints: cfg_scale=1.0, and the docs state cfg_scale and negative_prompt "have no effect on post-trained checkpoints". When CFG is active (cfg_scale ≈ 7 on a -base checkpoint, or --cfg 3.0 in the TRT runtime) it is batched into a single forward at doubled batch, doubling both compute and activation memory; the TRT engine is static batch 1 and implements it as a sequential cond+uncond dual pass instead.

Chunked decoding is on by default, using 256-latent windows with overlap, overridable per call via chunked_decode=. It is a memory mechanism, not a speed one — the vendor's TRT notes are explicit that single-shot is 10–20 % faster above L=256 — but on a 4 GB card it is what makes a long render possible at all.

Optional input encoding. Audio-to-audio and inpainting additionally run the SAME encoder over the init clip. Pure text-to-audio never touches it.

What did not fit: medium

medium (1.4 B) was downloaded and attempted. It fails on host RAM, not VRAM, and it never reached the GPU.

The mechanism is in load_diffusion_cond (stable_audio_3/loading_utils.py):

model = create_diffusion_cond_from_config(model_config)
copy_state_dict(model, load_file(ckpt_path))   # whole 9.22 GB checkpoint
model.to(device).eval().requires_grad_(False)

load_file() materialises the entire 9.22 GB checkpoint and copy_state_dict then builds a second state dict to copy into, so peak host usage is a multiple of the checkpoint size before any GPU transfer. On a 7.5 GB machine the kernel kills the process.

What the record actually shows. The download completed cleanly: 17/17 files, 9.8 GB on disk, DOWNLOAD_EXIT=0, 32 minutes. The kernel then logged two OOM kills of the same Python process, at 18:20:44 and 18:32:07, with total-vm 31.5 GB and 28.8 GB respectively. Note the accounting: anon-rss in those lines is ~0 and file-rss ~3 MB, because the memory was held as page cache over the memory-mapped checkpoint rather than as anonymous mappings — the kernel reclaimed it and still had to kill the process. There is no generation output for medium: no JSON sidecar, no WAV.

The article's earlier draft reported this as "RSS climbed to 7,441 MB" and "exit code 137"; neither figure appears in any log or journal entry, and the kernel's own accounting says total-vm reached 28.8–31.5 GB. Treat the earlier numbers as retracted — the OOM kills are real and timestamped, the peak-RSS figures were not sourced.

Related correction: the weights are not "under 3 GB". The 433 M parameter count is the DiT alone. medium's total is DiT 1.4 B + SAME-L 852 M + T5Gemma 284 M, i.e. 5.07 GB of weights in fp16 before any workspace, which is why the "under 3 GB fits in 6 GB" framing was never the right question. (SAME-L's shipped model.safetensors is 3408.6 MB fp32 → 852 M parameters; the model overview's claim of 1.7 B looks like an fp32 byte count read as an fp16 parameter count.)

Decision: medium is not a target for this laptop. Port it to the TPU / large-GPU side of the project instead.

Precision: what actually runs

The checkpoint ships float32 and the library casts everything to float16 at load. All 685 tensors in small-music/model.safetensors are F32, and StableAudioModel.from_pretrained(model_name, device, model_half=True) casts the whole model down. Measured dtypes on victoria after load:

The decoder therefore runs fp16, not fp32. An earlier version of this article stated that SAME-S should stay fp32 to avoid the differential-attention cancellation the vendor documents, costing 0.43 GB. That is a recommendation the shipped default does not follow, and the default is what all the numbers above were measured under. The vendor's warning is about the fp16 conversion of the TensorRT and MLX decoder paths; the PyTorch path casts the autoencoder to fp16 unconditionally and does not surface a warning. Whether the 120 s output actually shows the documented degradation was not tested here.

A whole-model dtype cast is available and changes both cost and footprint. inner.to(dtype) on the loaded model, measured with the GPU clock already ramped (bf16 is supported on this GPU — torch.cuda.is_bf16_supported() is true). Steady state, mean of calls 2–3, seconds:

Stage-level:

bf16 and fp16 are indistinguishable in both time and memory — every fp16/bf16 pair is within 6 % and the sign of the difference flips between durations, which is what the sm_86 rate equality predicts. fp32 costs 37 % more peak VRAM and ~2.5× the time, concentrated in the DiT (3.902 vs 1.416 s at 120 s, 2.76×) more than the decoder (1.400 vs 0.734 s, 1.91×). Numeric differences between the three were not measured, so nothing here speaks to accuracy.

Why fp32 kernels appear in an fp16 trace. The decoder's 114 parameters and 6 buffers are all float16 and its output is float16, yet the 120 s trace contains vectorized_layer_norm_kernel<float> and AUnaryFunctor<float> slices. These come from regions deliberately excluded from autocast rather than from fp32 weights: stable_audio_3/models/blocks.py and transformer.py decorate specific forwards with @torch.amp.autocast("cuda", enabled=False) and cast to .float() inside — the timestep and Fourier-feature embeddings, and the transformer's normalisation path. So a Half and a float variant of the same kernel coexisting in the trace is expected, not a dtype leak.

Four backend flags are set unconditionally at import (stable_audio_3/model.py):

torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False
torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction = False
torch.backends.cudnn.benchmark = False

The third pins fp16 GEMMs to the fp32-accumulate rate; the fourth disables cuDNN autotuning. generate.py exposes --enable-tf32 and --enable-fp16-reduction so each can be measured. The cold path also pays a one-time 17–22 ms cost in one DiT block, which is the signature of kernel autotuning; the flags have not been swept. Since the model runs fp16, allow_fp16_reduced_precision_reduction = False is the one that applies to the observed workload.

Why the decoder costs what it does

The decoder is 33.9 % of the warm 120 s request — 0.734 s against the DiT's 1.414 s across eight steps. It is not one slow kernel; it is a large number of small ones.

It runs 14 sequential chunk decodes with 39 % redundant work

decode_audio(latents, chunked=True) defaults to chunk_size=128 and overlap=32, both in latent units. For 120 s (1292 latent frames) the chunk starts are 0, 96, 192, … , 1152 plus a final window anchored to the end at 1164:

Each chunk is a separate decoder invocation producing 524,288 samples (11.9 s of audio), and 39 % of the arithmetic is thrown away in the overlap trim.

Single-shot decode is 32 % faster, and it fits

Decoding the same 1292-frame latents directly, measured on victoria:

Single-shot is 1.47× faster (0.499 s vs 0.734 s) for identical output length, and peak VRAM is unchanged at 2179 MiB — it did not exceed the card. The vendor's TensorRT notes put single-shot scratch at ~5.5 GB, but that is the TRT engine's workspace; the PyTorch path on this card did not allocate anything like it. So the chunking is not buying memory here, and it is paying 47 % more time for 39 % more work.

Where the decoder's device time goes

From the Perfetto trace of a warm 120 s call (same_decode range: 738.7 ms wall, 734.9 ms device, 5,672 kernel slices):

Device time is 99.5 % of the range's wall time, so the decoder is not idle-waiting: it is genuinely busy for 735 ms. Elementwise, copy and normalisation kernels together outnumber GEMM calls 14:1 and outweigh them, at a mean of 50–270 µs each. That is the signature of many small tensors rather than one large one: 14 decoder passes over 128-latent windows means every layer in the stack runs at a token count of a few hundred, and the per-kernel launch and tail overhead is paid 14 times over.

The DiT, by the same measure, for comparison

Over the eight steps of one thread (1,382.3 ms device across 8 × 173.7 ms; 13,472 kernel slices):

The class shares are identical on both worker threads to three significant figures (47.8 / 43.7 / 8.5), so the mix is a property of the model rather than of which thread is counted — only the absolute call counts and device totals differ between them. The DiT is more GEMM-weighted than the decoder (47.8 % vs 34.8 %) and its elementwise share is lower (43.7 % vs 58.9 %), but both stages spend more than 40 % of device time outside the tensor cores.

Per step, one thread's DiT work is 173.7 ms of device time and 1,704 kernel slices — about 9.8 µs per slice, which is short enough that launch and tail effects are a plausible part of the cost rather than a rounding error.

Top kernels, whole warm 120 s call

Every kernel slice in the trace, both worker threads included: 20,108 slices, 2,127.9 ms total device time, against a recorded wall time of 2,213.5 ms. The two threads overlap, so this is a mix of both; the per-stage sections above each use one thread.

Three observations the numbers support without interpretation: the two 160x128x32 and 192x128x32 GEMM tile shapes alone are 34.5 % of all device time; flash_fwd_kernel runs with a head dim of 64 and is 7.5 %; and vectorized_layer_norm_kernel appears in both Half and float variants, 117 ms combined — so some normalisations are not in the model dtype.

Measurement conditions

Reproducing these numbers depends on two things that are easy to get wrong on this host, and both were hit during this pass.

• The GPU idles at 210 MHz and ramps under load. nvidia-smi reports clocks.sm 210 MHz idle against a 1490 MHz maximum. A short measurement taken before the ramp completes reads roughly 2× slow across every stage uniformly — an early dtype sweep in this pass produced a 30 s DiT of 1.47 s that fell to 0.69 s once the card was warm, with decode scaling identically. Numbers here were taken with the clock already ramped; scripts/gpuwatch.py samples clocks alongside a run so a timing can be labelled with the clock it was measured at. Compare only measurements taken at the same clock state.

• Victora is a shared machine and concurrent GPU work invalidates a measurement. A sustained-bandwidth soak that read a flat 127.83–127.85 GB/s for 45 consecutive windows collapsed to 59–73 GB/s in three later stretches with memory and SM clocks unchanged and power rising to 37–46 W. The journal shows three concurrent short-lived sessions running GPU work in those same minutes, so it was contention, not throttling. Under contention, bandwidth fell by ~45 % and every timing in this article would be wrong by a similar factor. Check nvidia-smi --query-compute-apps before trusting a run.

• The GPU does not throttle under sustained load. A 90 s read soak held the memory clock pinned at its 5486 MHz maximum and the SM at 1702–1725 MHz, with temperature 53→60 °C at ~31 W. The flat 128 GB/s figure is a sustained number, not a burst.

• torch.profiler inflates wall time without changing the ranking. The traced 120 s call shows 174 ms per DiT step against an untraced 177 ms, so inflation is small here; the decode figure is likewise nearly unaffected (713.8 ms traced vs 734 ms steady state). Kernel shares and per-kernel means from the trace are sound, and the per-step shape — including the step-1 anomaly — is only visible in the trace.

• The binding constraint on this host is disk and host RAM, not compute. The card used 2179 of 6144 MiB; the machine ran out of host memory loading medium, and the 9.8 GB download took free space from 24 GB to 15 GB. The GPU is the least contended resource in the system.

• flash_attn does not work here as configured. A cached flash_attn-2.8.3 wheel installs, but its .so was built against torch 2.7.1 and fails under 2.13 with an undefined symbol (c10_cuda_check_implementation). small-music does not need it; medium does, and falls back to the library's own SDPA cascade rather than raising. The trace nevertheless shows pytorch_flash::flash_fwd_kernel running at 7.5 % of device time, so the SDPA cascade is selecting a flash kernel from somewhere.

• small-music runs fully offline. A complete checkpoint already existed locally (2.27 GB model.safetensors + 1.18 GB t5gemma-b-b-ul2/model.safetensors), so no download was needed for the runs above.

• torchaudio.save() needs torchcodec, which is not a declared dependency; generate.py writes through soundfile instead.

• Per-request cost is 2.04 + 0.0172 × duration cold and 0.29 + 0.0163 × duration warm. The two differ almost entirely in the constant. A first request in a fresh process costs 1.4–2.1 s more than every request after it.

Corrections to the primary sources

Three discrepancies found reading the release against its own artifacts. Each is independently checkable and each materially changes a footprint estimate.

1. Latent count per second is misstated by 2× in the model overview — it claims 216 latents for a 10 s clip; the correct value is 108, i.e. 10.7666 latents/s at 4096 samples per channel. Confirmed three ways (TRT L-range endpoints, the "L=8192 is 761 s" note, and the SAME-S decoder ceiling of L=1292 = exactly 120 s).
2. Autoencoder parameter counts do not match the shipped weights. The doc says SAME-S 266 M and SAME-L 1.7 B; the artifacts give 108 M and 852 M. SAME-S is confirmed twice to four digits — the MLX fp32 encoder and decoder files sum to 433.0 MB, exactly the size of stabilityai/SAME-S's model.safetensors. The 1.7 B figure appears to be an fp32 byte count read as fp16 parameters. Possible that the doc figures are training-time totals including the SAME paper's diffusion-alignment model and discriminators; unconfirmed.
3. The small model's maximum duration is stated inconsistently across the README (380 s for medium) and the model-overview table (medium ~4.75 min, large ~6.3 min). The duration conditioner clips at 384 s, which supports the README. Not resolved.