← All notes and experiments

Making a ternary reasoning model fast in vLLM

Earlier development stage. Implementation speeds here describe this stage of the work. See the latest published comparison for the current checkpoint.

I wanted to serve PrismML's Ternary Bonsai 2 27B to several users from an RTX A4000 with 16 GB of VRAM. llama.cpp already supported the model. I wanted to bring it to vLLM for its ability to batch work and serve concurrent requests, but vLLM could not run it when I started.

The first job was to add support for the model's PQ2 weight format. That produced a working implementation at about 25 tokens per second on my RTX 3090 Ti development card. From there, the question became how to make generation faster, share the GPU efficiently across users and leave enough memory for their conversations.

This series follows the work from October 2026: what helped, what failed and what the measurements taught me. First, how does a model this large fit in so little memory?

Measurement note. Development results come from one RTX 3090-series GPU at a time, identified with each benchmark. The A4000 is the 16 GB deployment target; its capacity estimates are labelled separately.

A model with 27 billion learned numbers sounds too large for that card. The answer lies in how it stores its weights and remembers earlier text.

A few terms will recur. A weight is a number learned during training. A token is a piece of text, often part of a word. Context is the text the model can use while producing its next token. VRAM is the GPU's memory. A kernel is a small program that runs on the GPU; much of this series is about making those programs faster.

The model description and published quality benchmarks in this post come from PrismML's whitepaper, linked from its official release page, the GGUF model card, and PrismML's release announcement. The model and format are theirs. Parts 1 and 2 explain the engineering problem; part 3 begins the account of my implementation and optimisation work.

The headline

FP16 stores each weight using 16 bits. The smaller formats on the right make the weights fit on the A4000, leaving some memory for the information saved while reading a conversation. That saved information is called the KV cache; we will return to it later. The FP16 parent needs 54 GB just for weights.

The more interesting claim is which abilities survive. Quantisation means representing numbers with fewer possible values to save space. The whitepaper compares Bonsai with another compressed version of the same parent model, llama.cpp's IQ2_XXS. That version uses 2.2 bits per weight (bpw) and occupies 7.3 GB:

Published quality check

Source: whitepaper Table 10, reasoning effort xhigh.

The whitepaper describes the failure of conventional low-bit quantisation as qualitative and uneven. Surface fluency and short tool calls survive. Long-chain reasoning, competition maths and code are what break. That makes the damage easy to miss in a quick chat and expensive to find in production.

Bonsai is smaller than IQ2_XXS but keeps those hard capabilities. That gap is why I think the model is worth serving well.

The format: three values and a scale

Most of the model's weights are stored in tables called matrices. Bonsai represents each entry as one of three choices, −1, 0 or +1, multiplied by a shared number called a scale. Each group of 128 weights shares one 16-bit scale:

A weight is therefore either the scale, its negative or zero. Within one group, it cannot take an arbitrary value such as 0.37 or −0.81. The pattern of signs and zeros has to carry information that a less compressed model can store in more precise numbers.

For a small example, take choices [−1, 0, +1, +1] and a shared scale of 0.5. The weights are [−0.5, 0, +0.5, +0.5]. Given input numbers [2, 3, 4, 1], multiply each pair and add the results: 0.5 × (−2 + 0 + 4 + 1) = 1.5.

This calculation is a dot product. The input numbers produced as text passes through the model are called activations. A kernel can read the compressed weights and do this calculation without saving a large, uncompressed copy.

How many bits is that?

A symbol with three equally likely values carries \(\log_2 3 \approx 1.585\) bits. Add one 16-bit scale shared by 128 weights:

A few arrays of numbers, called tensors, retain more precision (see below). Including them brings the language model to 1.72 bits per weight. That is a 9.3× reduction from FP16.

Hardware doesn't deal in fractional bits, so real files use a packing. PrismML ships two:

PQ2_0 uses about 0.4 extra bits per weight because two bits allow four choices and the model uses only three. In exchange, the GPU can recover each choice with a few simple operations on bits.

PTQ1_0 gets close to the limit with a denser trit encoding. As an illustration (not necessarily PrismML's exact scheme), five trits fit in one byte because 3⁵ = 243 ≤ 256, giving 1.6 bits per weight. The cost is more arithmetic to recover each value.

PQ2_0 is the format I work with throughout this series. Part 2 dissects its byte layout.

Why ternary rather than binary?

The predecessor, 1-bit Bonsai, used {−1, +1}. Adding zero matters for two reasons:

1. It lets the model say "this input doesn't matter here." A binary weight must push every input up or down. A ternary weight can ignore an input entirely, so the zeros act as learned sparsity.
2. It improves rounding. With zero available, small-magnitude weights round to 0 instead of being forced to ±s. This removes a large, systematic error on the many near-zero weights in a typical layer.

Representing three choices rather than two requires about 0.585 more bits per weight in theory.

Why so little is lost

Directly rounding a conventional model's FP16 weights to {−s, 0, +s} would substantially change its calculations. Bonsai uses three approaches to preserve useful model performance at this precision.

1. Mix the numbers before compressing them

Some input values are much larger than others. If one large value determines the scale for a whole group, smaller values can lose most of their detail when rounded. Bonsai first mixes the input values so those large values are spread across a group. The weights are represented in this mixed coordinate system, called a rotated basis.

The particular mixing operation is a Hadamard transform: repeated additions and subtractions, followed by scaling. The formula below describes the rotation and the following weighted sum:

• \(H_n\) is a Walsh–Hadamard matrix: entries are ±1, and its rows are mutually orthogonal.
• \(S\) is a fixed diagonal of random ±1 signs.
• Scaled by \(1/\sqrt{n}\), \(R\) is orthogonal, so it rotates vectors without changing their length.
• It is applied blockwise, to each 1024-element chunk of the activation.

An outlier is a value much larger than the others in its group. A few input positions can carry values tens of times larger than the rest. Without mixing, one outlier can set the scale for the whole group and cause smaller values to round towards zero.

The transform mixes each group of 1024 input values using additions and subtractions. This spreads the effect of an outlier across many positions, making a shared scale more useful.

This idea comes from the rotation-based quantisation literature: the whitepaper cites SpinQuant; QuaRot and QuIP# are in the same family.

The rotation itself preserves lengths and can be undone. If weights and inputs are changed consistently, this change of coordinates preserves the calculation before quantisation. Rounding the weights to three choices is still an approximation.

The cost is extra work during generation. For 1024 values, the transform needs 10 rounds of additions and subtractions. The arithmetic is small compared with the following matrix multiplication, but starting another GPU program takes time too. Part 3 measures how much this matters.

The software mixes the input numbers, then calculates with the compressed weights. It does not create a full uncompressed weight matrix.

2. A few tensors stay in full precision

Bonsai 2 keeps exactly 26.2 M parameters (0.0976%) above the ternary format: unrotated and unquantised (whitepaper Table 2). They are:

• The recurrent state path of the linear-attention layers: in_proj_a, in_proj_b, conv1d, A_log, dt_bias and the gating norm. These control how the Gated DeltaNet state decays and updates. A small error there compounds over every token of a long context.
• All RMSNorm weights: small vectors that provide precise per-channel scaling.

The previous Bonsai release ternarised these as well. Keeping them precise costs 52 MB and moves the format from 1.71 to 1.72 bpw. The whitepaper ties this release's improvements in long-horizon agentic tasks to the model as a whole, rather than attributing them to this choice specifically.

3. The model is trained for this format

Post-training rounding to three levels would not produce these numbers. The weights are produced as ternary, so the network has learned to express itself with signs, zeros and group scales.

The whitepaper says very little about the training recipe itself. I won't guess beyond "it is a ternary model, not a rounded FP16 model". For my purposes, what matters is that the shipped weights are exactly \(s_g \cdot t_i\) in the rotated basis. The runtime must reproduce that arithmetic faithfully and nothing more.

How the model remembers earlier text

The model processes text through a sequence of layers, each transforming the numbers passed to it. Some layers use full attention: the current token looks back at information saved for earlier tokens. These layers need a KV cache that grows with the conversation.

Most layers instead update a fixed-size summary of earlier text. This is the model's linear attention, implemented with Gated DeltaNet. Using both kinds is called hybrid attention. Only 16 of 64 layers need a growing cache; that distinction will matter when we make room for longer conversations.

Three-quarters of the layers carry a fixed-size recurrent state instead of a growing KV cache. That is what makes a 262K context feasible on a small device, since only 16 layers keep per-token K/V.

It also creates practical headaches: most existing low-bit kernel stacks assume a plain transformer. As the whitepaper notes, PrismML had to write custom kernels for this hybrid structure on both Apple Silicon and CUDA. vLLM had the same problem, as we'll see.

What the benchmark scores leave open

• Reasoning effort matters. At medium effort, Bonsai's average drops to 79.3 against FP16's 82.6 (whitepaper Table 11). The gap widens on AIME (74.6 vs 86.3 on AIME25). The model seems to lean on its thinking budget to recover precision. That makes decode throughput especially important: a reasoning model that thinks for 20K tokens per answer spends almost all its wall-clock time generating.
• Agentic long-horizon tasks keep roughly three-quarters of full-precision performance on Terminal-Bench 2.1 (52.8 vs 69.7) and SWE-bench Verified (60.8 vs 80.6). That is strong for 6 GB, but not parity.
• All quality numbers above are PrismML's, measured with vLLM on H100s. My own quality checks in this series are much smaller validation checks, and I'll label them as such.

Why this sets up the engineering problem

A model this small is only useful if the runtime keeps it small and makes it fast. Specifically:

1. Avoid a full floating-point copy of the weights. Expanding the whole model would throw away the memory advantage. Recovering a small tile in registers or shared memory while calculating is a different matter.
2. Start with a memory-traffic model for decode. A first estimate assumes each used weight is read once per token. That gives a bandwidth ceiling; measurements must establish whether memory traffic, unpacking or other work is the actual limit.
3. Pay for the rotation cheaply. It is a fixed cost per projection, and it matters most at batch size 1.

PrismML's own CUDA stack, built on llama.cpp, reaches 90.9 tok/s for PQ2_0 on an RTX 4090 and 142.5 tok/s on an RTX 5090 (whitepaper Table 5, tg128). The vLLM GGUF plugin had no PQ2 support before this work. After adding an initial implementation, I measured about 25 tok/s on my RTX 3090 Ti. That first working path is the baseline for the optimisation story.

The weights fit. The next question is how much data the runtime must move to produce one token. Part 2 turns that into a byte budget and a rough speed limit, giving us something to test the first implementation against.

For a visual explanation of GPU workers and number formats, see the concepts companion.

Sources

• NVIDIA RTX A4000 specifications: the 16 GB deployment target and its 140 W power budget.

• PrismML whitepaper (official release page): PrismML, Ternary Bonsai 2 27B, September 2026. Tables 1–3 (format, footprint), 7–11 (benchmarks), §2.4 and Appendix A.2 (rotation).

• Qwen3.8-27B config.json: model dimensions and layer pattern.

• Rotation and low-bit quantisation references: includes SpinQuant and the source model references.

• llama.cpp tensor encoding schemes: GGUF packing terminology.