Making a ternary reasoning model fast in vLLM
How can a compressed reasoning model serve several users on a memory-constrained GPU? This investigation follows the work of adding model support to vLLM, then improving kernels, batching and conversation memory.
Published measurements are snapshots of the stated hardware, settings and revisions.
I wanted to serve PrismML's Ternary Bonsai 2 27B to several users from an RTX A4000 with 16 GB of GPU memory. I chose vLLM for its ability to batch work and serve concurrent requests. When I started, vLLM could not run this model, so the first job was to add support for its PQ2 weight format. The next was to make that support fast enough to be useful.
The model stores most of its 27 billion learned numbers using just three choices: −1, 0 and +1, with a shared scale. This saves space, but the software has to calculate with that compressed representation efficiently. This series follows the work from the first working implementation through faster kernels, concurrent serving and a smaller conversation cache.
The code I developed for this project was co-authored with OpenAI Codex & Claude.
Current checkpoint
The refreshed comparison uses one RTX 3090 Ti at 300 W, with 8-bit KV caches, an FP32 recurrent state and no speculative decoding:
| Workload | Result | Scope |
|---|---|---|
| One request, plain decode | 94.4 tok/s | 1.42× llama.cpp in this comparison |
| Eight concurrent requests | 451 tok/s aggregate | 2.82× llama.cpp; total output across users |
| Prefill, 1,024–16,384 prompt tokens | 1.52–1.60× llama.cpp | Includes the output-projection and CUDA prefill changes |
| Prefill, 128 prompt tokens | 0.99× llama.cpp | The final refresh is close to level |
Using BF16 recurrent state raises plain decoding to 96.5 tok/s for one request and 493 aggregate at eight. Its short-task and prediction-loss checks are encouraging; long-reasoning validation is still incomplete. This state option is separate from the INT8 conversation cache.
The updated twelve-prompt speculative measurement averages about 210 tok/s at 1K prompts and 181–194 tok/s at 4K, depending on state precision. Speed varies substantially between prompts. These results supersede the earlier small-sample figures as the current speculative summary.
Chapter 9 explains the recorded stages, comparison settings, quality checks and compile-cache findings. llama.cpp's recurrent-state checkpoints are disabled for prefill. These records were integrated on 16 October 2026; the website build does not rerun GPU experiments.
How to read the results
Chapters 1–8 follow earlier implementation stages. Chapter 9 brings the comparison up to date. Compare numbers within the stated workload and revision; gains from different stages do not add together.
| Measurement type | What it tells us |
|---|---|
| Component test | How fast an operation or replay runs. It leaves out some work in a complete request. |
| Request performance | The user’s wait or the server’s output rate. Aggregate tok/s adds output across users. |
| Quality check | Numerical agreement or performance on a set of questions. Small numerical changes and similar task scores are different kinds of evidence. |
The precision settings apply to different parts of the model:
| Setting | What changes | Main trade-off |
|---|---|---|
| PQ2 weights | How the learned weights are stored; most use ternary values | A smaller model, with unpacking work during calculation |
| INT8 KV cache | How attention stores information from earlier tokens | Less memory per token, with conversion work and numerical changes to check |
| FP32 or BF16 recurrent state | How DeltaNet stores its running state | BF16 reduces state traffic; long-reasoning validation remains incomplete |
The recurrent state and KV cache are separate. Changing one does not change the other. Intermediate calculations can also use their own precision.
Choose a reading path
- Follow the engineering story: read chapters 1–9 in order, from model support to the current results. The bottleneck moves from arithmetic and data layout to power, attention and memory.
- See where the implementation stands: start with chapter 9.
- Run the model: use the installation guide, a companion to the investigation. The recipe pins the public revisions; benchmark settings are documented separately.
- Understand the implementation: start with GPU concepts and data flow, then browse the 39 recorded component experiments.
Worked examples introduce the arithmetic. Expandable sections keep the code, detailed measurements and historical settings available alongside the main explanation.
Measurement hardware
Development measurements use one RTX 3090-series GPU at a time. Each benchmark identifies the card, power limit and workload needed to interpret it. The 16 GB A4000 remains the deployment target; its capacities are estimates until measured on that card. The series compares software changes within matched runs.
Development setup and deployment target
The development machine has two RTX 3090 Tis and an RTX 3090. They share the Ampere architecture with the A4000, while memory capacity, bandwidth, multiprocessor counts and power limits differ. Shared instructions make the techniques relevant; deployment measurements must establish the target card's operating point.
Chapters
- A 27B reasoning model in 1.7 bits per weight
What ternary weights are, and how rotation, a few full-precision tensors and ternary-native training retain 98.2% of the reported 20-benchmark average
- From model size to GPU performance
A byte budget and bandwidth ceiling for decode (148 tok/s), the PQ2_0 block layout, and a checklist for the ideal kernel
- Establishing a baseline and using integer arithmetic
Measure the 2–4× gap to llama.cpp, then reduce it by using small integers and arranging the weights for faster reads
- Improving single-request generation
Repeat calculations with real model inputs, recover weights with fewer instructions, and write a program just for one token
- Batching, kernel performance and power limits
A trick that loses at batch 1 and wins at batch 8, why promising kernel gains disappeared in serving, and what power and clock measurements revealed
- Long-context attention: balancing latency and throughput
Why attention grows with context, and how using different kernels for prefill and decode avoids a latency trade-off
- Small batches and speculative decoding
Make small batches efficient, use them to verify draft tokens, then preserve the gain under load and at long context
- Reducing memory use with an INT8 KV cache
Remove wasted cache space, fix a memory-overwrite bug, and nearly double saved-history capacity while checking speed and answer quality
- Current results: comparing vLLM and llama.cpp
Refreshed serving results, three kernel rounds, BF16 recurrent state, broader speculative measurements and corrected compile-cache comparisons
Companion guides
- GPU concepts and data flow
Understand tiles, layouts, numerical checks and the data flow behind the experiments.
- Running the model locally
Install and launch the model using a pinned recipe, with optional DFlash support.
- Kernel experiments: recorded evidence
Browse all 39 component-test records, with diagrams and expandable setup and code.
Code and sources
PrismML created the model and its formats. I added the initial PQ2 support to the vLLM GGUF plugin, then developed the kernels, serving changes and measurements described here.
- Model: GGUF model card, MLX model card and PrismML's official release page.
- Implementation: my public PQ2 plugin fork and the vLLM fork with DFlash and quantised-context changes.
- Background: GGUF quantisation formats, vLLM's GGUF documentation, A4000 specifications and Ampere tuning guide.