← All notes and experiments

Making a ternary reasoning model fast in vLLM

Earlier development stage. Implementation speeds here describe this stage of the work. See the latest published comparison for the current checkpoint.

After part 4, single-request decode was about 72 tok/s. I expected smaller kernel improvements to carry it further. Instead, a set of candidates that gained 2.6–3.0% in linked replay changed serving speed by only 0.08%.

The question on 6 October was simple: why did a faster kernel barely change request speed? The answer involved the GPU’s power governor.

A separate eight-request experiment brought another lesson: a change that lost on one workload could win on another.

Measurement note. Unless stated otherwise, experiments use one RTX 3090 Ti at a 275 W cap. The power sweep deliberately varies that limit. The complete-step replay comparison uses an RTX 3090 and is labelled separately. These are comparisons of software changes within each experiment.

Start with the promising ideas

Several plausible changes had already failed: deeper pipelines, prefetching, wider layouts and fusion. They were useful clues about the limits of the current kernel. The detailed outcomes are retained below.

Combining steps can also change the answer slightly. I rejected the first twelve combined programs because their numerical error was too large. Adding floating-point numbers in a different order changes rounding. That changed some of the integers produced by the next rounding step and exceeded my 0.2% error limit, measured by relative root-mean-square error. Versions that preserved the reference's summation order matched all 768 real-input checks exactly.

I also ran an all-reference control: the same kernel as both "baseline" and "candidate". It showed apparent gains of about 1% in some pairs, which is a useful calibration for what counts as noise.

What the GPU was waiting for

The profiling tool reports how GPU workers spend their time. A thread is one worker; a warp is a group of 32 threads scheduled together. Occupancy measures how many groups are present relative to the hardware limit. They can be present but still waiting for data.

The table uses the tool's names. DRAM is the card's main memory. A long-scoreboard stall is a wait for a memory operation. No eligible warp means there is no group ready to run. High occupancy helps only if some of those groups can do useful work while others wait.

Why did none of the latency-hiding ideas work? Nsight Compute counters had been blocked (ERR_NVGPUCTRPERM) until I enabled profiling for non-admin users. Once they were available, they told a clear story for the six production single-token kernels:

• Long-scoreboard stalls dominate. Warps spend 46–67% of their cycles waiting on global memory loads.
• Schedulers are often idle. In 42–58% of cycles no warp is ready to issue.
• DRAM is busy but not saturated. It runs at 68–78% of peak on the decoder projections, versus 90% on the LM head.
• Registers cap occupancy at 30–65%.
• Integer ALU utilisation (40–65%) is below DRAM utilisation everywhere, which suggests unpacking arithmetic alone offers limited scope for improvement.

The counters suggest memory latency is limiting these decoder projections. There aren't enough loads in flight to keep DRAM saturated. In principle, more resident warps or more independent loads per warp can help. Both consume resources, which explains why simply adding prefetches can make the kernel worse. The LM head is the only kernel close to the bandwidth wall in this profile.

Try to do useful work while reads are waiting

I then tried 126 variants that start more reads early or make room for more workers. There is a cost: keeping more work in progress needs more temporary storage. Limiting registers too far makes temporary values spill into slower memory. Tested on their own, five choices improved particular matrix sizes:

Prefetch and extra stages lost again, and register caps spilled again. The linked chain improved 2.6–3.0%.

In serving with automatic clocks, the result was +0.08% (70.896 → 70.953 tok/s). Essentially nothing.

The card changes its own clock speed

The power governor is the control that adjusts the GPU's operating speed to stay within its limits. A higher clock means more cycles per second, but the speed also depends on how much useful work each cycle does.

A faster fixed-clock kernel can deliver a smaller gain under automatic clocks. The direction and size of the clock change depend on the workload and card.

The replay kept promising a gain, but complete requests barely moved. Then I checked the clocks: the candidate ran at a median busy clock of 1380 MHz, compared with 1425 MHz for the reference. The faster kernel was running on a slower clock.

At the 275 W cap, this workload sat right against the limit, at about 273 W during decode. A kernel that keeps more memory requests in flight draws more power per unit time. The governor responds by lowering clocks, and that cancels the gain.

Earlier improvements in this series (geometry v2, single-token kernel) had shown the opposite effect. Leaner kernels let the clock rise (1395 → 1455 MHz), which contributed to their measured serving gains.

To separate the two effects I ran a fixed-clock test: the GPU locked at 1350 MHz, 275 W limit unchanged, the same five latency-hiding kernels against the reference.

Request performance

The kernel gain is real. At the power cap, it is spent on power rather than speed. On a power-limited consumer card, performance per watt is the objective, and "faster at the same clock" is only half the story.

This has methodological consequences for everything else in the series:

• Gains at automatic clocks mix kernel speed with power efficiency. They are real for these development-card measurements, but they don't transfer as kernel-only claims.
• Small kernel improvements can vanish, or reverse, in serving, for reasons unrelated to correctness or code quality.
• I now record clocks and power alongside every serving number, and treat a fixed-clock A/B as a separate, explicit experiment.

How much speed the power limit was costing

If the power control was cancelling small kernel gains, how much did the chosen power limit matter? I had capped the development card at 275 W. The following experiment varies only that limit on the same GPU.

I kept one server running and changed only its GPU power limit, running three 4096-in / 2048-out requests per level:

Request performance

300 W ran in a separate session. Repeating 275 W at the end of the main sweep reproduced the first result within 0.7%.

Three things stand out.

The power setting had a substantial effect on throughput. An additional 25 W gave about +12%, and 50 W gave about +18%. The latter was comparable to the single-token kernel improvement in part 4, without a software change.

Energy per token has a minimum near 300 W. Below it, fixed costs (memory, board, leakage) are spread over fewer tokens per second. Above about 325 W, every extra MHz needs more voltage, so power climbs much faster than speed. At 450 W the card doesn't even use the headroom: it tops out at about 430 W and 2055 MHz.

The memory clock never moved (10376 MHz at every level). All of the speed-up came from the core clock: +32% SM clock bought +18% tok/s. A purely bandwidth-bound kernel wouldn't care about the SM clock. This is the stall-counter diagnosis from earlier, measured a different way: decode in this run is still limited by latency and instruction issue, not by DRAM.

Even at 86 tok/s the projections move only about 585 GB/s, so the ceiling from part 2 is still a long way off.

For later experiments on this card I used 300 W: a useful speed increase without the much higher energy cost of the largest limits. The deployment card needs its own power and throughput measurements.

A separate lesson: batch size changes the trade-off

This experiment used the earlier 275 W configuration, before the power-limit change above. It asks about eight requests sharing each weight read.

Eight requests can generate their next token together. That gives the GPU eight rows of input numbers to calculate against the same weights, written as M = 8 in the code. Ideally it reads each piece of the weights once and uses it for all eight rows. My integer program already did that. Why was there still so much work?

The first checks were negative. My v2 tiles from part 4 only apply at M = 1 (+0.97% and −0.54%, i.e. nothing). A fresh 260-configuration geometry screen at M = 8 found no tile that cleared the noise threshold. Interleaving token groups for better L2 reuse didn't reliably help either.

The trace pointed somewhere else:

GPU profile

The output projections at M = 8 were running through the floating-point prefill kernel with a 64-row request tile. Eight real rows were padded out to 64, so seven-eighths of its work was waste.

Two fixes, both bit-exact

1. A smaller float tile. BM = 8 for that shape at M = 8 made the output projection 2.18× faster (7.560 → 3.338 ms in replay) and the linked chain 9.2% faster. It is bit-exact, because each output is computed the same way, just without the padding.
2. Chained DP4A. This approach was 13% slower at M = 1 in part 4. At M = 8 it was 3.6–4.0% faster in replay, also bit-exact.

At M = 1 the bottleneck is getting weight bytes in, and independent DP4As give the scheduler more freedom. At M = 8 each weight word feeds eight activation rows, the kernel does eight times the integer arithmetic per byte, and chaining the accumulator saves instructions and registers where they now matter. The same idea can be right or wrong depending on batch size, so defaults must be scoped by batch size.

Think of four independent dot products as four jobs that can be scheduled while other jobs wait. Chaining them into one accumulator makes each update depend on the previous result. That can reduce temporary storage, but it also removes scheduling freedom. The batch-one result favoured independence; batch eight changed the balance between arithmetic, storage and reuse.

A third candidate, sending the output projection through the integer path at M = 8, was worth another 10.6–11% in replay. But it changes activation quantisation (relative RMS 0.0085), so I left it out.

Serving

Eight synchronised requests, 90-token prompt, 2048 forced outputs, median of three:

Request performance

Single-request throughput was unchanged (−0.6% to +0.9%), as expected, since both rules only fire at M = 8. These became scoped defaults in the integer dispatch (GGUF_PQ2_BATCH8_CHAINED=1, GGUF_PQ2_BATCH8_FLOAT_OUTPUT_BM=8).

These are aggregate rates. For eight synchronised requests, 182 aggregate tok/s corresponds to roughly 22.8 tok/s per request. Higher total throughput can serve more users while each user still sees slower generation than in the single-request case.

llama.cpp at batch eight

I also traced llama.cpp on the same eight-request workload under the same Nsight settings:

llama.cpp's matrix-vector kernels are still faster than mine. Its total time is longer because of idle gaps, plus 7.5 s of CUTLASS BF16 GEMM and 5.9 s of row gathers absent from the vLLM trace.

So for batched serving, vLLM's scheduler and CUDA graphs are worth more than llama.cpp's better kernels. The kernels are still the target: integer GEMV plus floating projections are 85.5% of vLLM's kernel time.

Repeat an entire token-generation step

The projection-only replay from part 4 ranked candidates well but sized them poorly. For example, a +2.6–3.0% linked-chain gain turned into +0.08% in serving.

So I extended the harness to replay the entire decode step with real state. Each replay runs three linked CUDA graphs: a recurrent-state reset for all 48 linear-attention layers, the full model graph, then the LM head plus sampler. Correctness is checked by exact hidden state and sampled token, and by mutating the input token to prove the graph reads it.

Replay estimate and request performance

That is within about 0.2 percentage points, and the first proxy I'd trust to size a change, not just rank it.

Compare the result with the memory speed limit

Part 2 put the weight-only ceiling at 148 tok/s (6.75 ms per token) on a 3090 Ti. The earlier production snapshot used in part 6 runs at about 71.7 tok/s with a 4K context, or 13.9 ms per token. This is a different experiment from the power-sweep rows above. The trace from part 6 splits that step into:

Measured profile and bandwidth estimate

To go meaningfully faster, the projections need to approach 85–90% of bandwidth like the LM head does. That would save about 2–2.5 ms. The roughly 3 ms of "everything else" needs to shrink through fusion that doesn't change numerics. And all of it has to happen within the power budget.

None of this is impossible. But the easy, large wins were integer arithmetic, prepared layout and specialisation, and they are already taken.

The important result was learning how to judge the smaller gains: replay, clocks and complete requests answer different questions. Batch size changes the right kernel choice, while the power limit changes how much of a kernel gain becomes serving speed.

The next bottleneck grows with the conversation. Part 6 follows attention as the cache gets larger, and asks how to improve generation without making users wait longer for the first token.

The reference includes the batch-eight experiments, preparation fusion and memory-wait diagnostics. Those component tests are distinct from this chapter's power and serving measurements.

Sources

Implementation and profiling references:

• triton/pq2_int_gemv.py: Batch-size choices and chained integer calculations.
• NVIDIA Nsight Compute: profiler and hardware-counter documentation.
• llama.cpp CUDA matrix-vector implementation: comparison implementation.