Running the model locally

Companion guide · Published 14 October 2026 · Updated 16 October 2026

Making a ternary reasoning model fast in vLLM

Published measurements are snapshots of the stated hardware, settings and revisions.

This is the practical companion to the investigation. It gets Ternary Bonsai 2 27B running on Linux, either directly on your machine or inside a container. The scripts are included below, and the model and source code come from public repositories.

The recipe targets an Ampere GPU such as the 16 GB RTX A4000 or a 3090. Start with plain decoding and a context length chosen to fit memory; add the helper model after the basic server works. The current results chapter describes the benchmark configuration separately. The scripts pin the public fork revisions checked on 16 October 2026. Their settings are a starting point, rather than a reproduction of the benchmark environment.

Choose your installation route

Local installation: build and run inside a Python environment on your machine.
Container: build the same software into a Podman image, using your host's NVIDIA driver.

Both routes use the same install.sh and serve.sh files. The sequence is: check prerequisites, create those files, build locally or in a container, download the model, then send a test request.

Before starting

You need an NVIDIA driver that supports CUDA 13.0, plus enough disk space for the model, source build and compiled programs. Check the GPU with nvidia-smi. For local installation, also install the CUDA 13.0 toolkit, Git, a C++ compiler, Python 3.12 with virtual-environment support, and uv. The container supplies the build tools, but the host still needs its NVIDIA driver. Both routes use uv on the host to install the model-download tool.

Create a working directory and run the following commands from it:

mkdir -p bonsai/models
cd bonsai

Create the installer

Save this as install.sh. It installs two fixed revisions of my forks: the PQ2 implementation and the DFlash context fix. It also installs PrismML's GGUF reader, which understands the PQ2 weight format.

cat > install.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail
uv venv --python python3 .venv
source .venv/bin/activate

git clone --branch dflash-quantized-context-kv https://github.com/akmorley/vllm-prism.git
git -C vllm-prism checkout d463094f1d97d95ac6c9596b92d3b7ce677f6fb9
git clone --branch pq2-bundled-clean https://github.com/akmorley/vllm-gguf-plugin-prism.git
git -C vllm-gguf-plugin-prism checkout 0121bbf2c34651f4bf89de082e79846e70291f08
git clone https://github.com/PrismML-Eng/llama.cpp.git llama

export TORCH_CUDA_ARCH_LIST=8.6
export VLLM_FA_CMAKE_GPU_ARCHES=86-real
export MAX_JOBS="${MAX_JOBS:-8}"
uv pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu130
uv pip install -r vllm-prism/requirements/build/cuda.txt
uv pip install --no-build-isolation ./vllm-prism
uv pip install --no-build-isolation ./vllm-gguf-plugin-prism
uv pip install --no-deps ./llama/gguf-py
SH

This builds the GPU programs from source and can take considerable time. Reduce MAX_JOBS if the compiler uses too much system memory. The two fork revisions are fixed; PrismML's reader and other dependencies follow the versions available when you install. This is a standalone source-build recipe, not the exact locked environment used for the benchmarks.

Create the launcher

Save this as serve.sh. Both installation routes use it. The server starts in text-only mode with the compressed weights and 8-bit conversation cache enabled.

cat > serve.sh <<'SH'
#!/usr/bin/env bash
set -euo pipefail
model_dir="${MODEL_DIR:-$PWD/models}"
export GGUF_PQ2_INT_GEMV=1 GGUF_PQ2_PREPARED=1 GGUF_PQ2_MMQ=1
args=(
  serve "$model_dir/Ternary-Bonsai-2-27B-PQ2_0.gguf"
  --tokenizer "$model_dir/qwen38-tokenizer"
  --served-model-name prism-27b
  --host "${SERVER_HOST:-127.0.0.1}" --port 8000
  --dtype bfloat16 --language-model-only
  --mamba-ssm-cache-dtype "${SSM_CACHE_DTYPE:-float32}"
  --attention-backend TRITON_ATTN --kv-cache-dtype int8_per_token_head
  --max-model-len -1 --gpu-memory-utilization 0.92
  --max-num-seqs 4 --max-num-batched-tokens 2048
  --reasoning-parser qwen3
)
if [[ "${USE_DFLASH:-0}" == "1" ]]; then
  args+=(--speculative-config "{\"method\":\"dflash\",\"model\":\"$model_dir/bonsai-dflash2\",\"num_speculative_tokens\":5,\"quantization\":\"fp8\"}")
fi
exec vllm "${args[@]}" "$@"
SH

The context length is set to -1, which tells this vLLM version to choose a length that fits the available GPU memory. The prompt and generated answer must both fit within that length.

Build using your chosen route

Local installation: run the installer, then activate its Python environment:

bash install.sh
source .venv/bin/activate

Container: install Podman and the NVIDIA Container Toolkit. Follow NVIDIA's GPU device setup, then check that nvidia-ctk cdi list includes GPU 0. Create this Containerfile beside the two scripts:

FROM docker.io/nvidia/cuda:13.0.0-devel-ubuntu24.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates curl git build-essential python3 python3-venv \
    && rm -rf /var/lib/apt/lists/*
RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
WORKDIR /opt/bonsai
COPY install.sh serve.sh ./
RUN bash install.sh
ENV PATH="/opt/bonsai/.venv/bin:$PATH" \
    MODEL_DIR=/models SERVER_HOST=0.0.0.0 \
    XDG_CACHE_HOME=/cache TRITON_CACHE_DIR=/cache/triton VLLM_CACHE_ROOT=/cache/vllm
ENTRYPOINT ["bash", "/opt/bonsai/serve.sh"]

Build your own image:

podman build -t localhost/bonsai-pq2 -f Containerfile .

The image uses NVIDIA's CUDA development image. It retains the CUDA toolkit because some programs compile on first use. It trades image size for a simpler setup.

Download the model and tokenizer

Install the Hugging Face download tool on the host with uv tool install huggingface-hub. Download the PQ2 model and the tokenizer, which translates text into token numbers:

hf download prism-ml/Ternary-Bonsai-2-27B-gguf \
  Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir models
hf download Qwen/Qwen3.8-27B \
  --include '*.json' '*.jinja' '*.txt' \
  --local-dir models/qwen38-tokenizer

For a local installation, start with:

bash serve.sh

For the container, start with:

podman run --rm --name bonsai27b \
  --device nvidia.com/gpu=0 --security-opt=label=disable --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$PWD/models:/models:ro" -v bonsai-cache:/cache \
  localhost/bonsai-pq2

The model files stay on the host. The container's named cache volume saves compiled programs between runs. Wait for the startup log to say the server is ready; the first start can take several minutes.

Check the server with a request

Both routes provide the same chat API:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "prism-27b",
    "messages": [{"role": "user", "content": "What is 17 * 23?"}],
    "max_tokens": 2048,
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20
  }'

The answer should be 391. Set max_tokens explicitly: 2,048 is useful for this first check, while harder reasoning may need a much larger budget.

Optional: use BF16 recurrent state

The launcher keeps the recurrent state in FP32 by default, matching the main column in the current comparison. To try BF16 locally, use SSM_CACHE_DTYPE=bfloat16 bash serve.sh; for the container, add -e SSM_CACHE_DTYPE=bfloat16 before its image name. This changes DeltaNet's fixed recurrent state, independently of --kv-cache-dtype int8_per_token_head.

The recorded tests found higher plain-decode throughput, small long-window prediction-loss changes and identical GSM8K answers in a 128-question cohort. GPQA and long-reasoning validation were not completed for this option. The pinned public plugin includes the later CUDA and ring kernels, but changing state precision alone does not reproduce the final comparison: its workload, kernel settings and power limit also matter.

Optional: add DFlash

The DFlash2 helper model suggests tokens that the main model checks together. Download its safetensors weights:

hf download ProCreations/Ternary-Bonsai-2-27B-DFlash2 \
  config.json model.safetensors LICENSE NOTICE \
  --local-dir models/bonsai-dflash2

Stop the plain server first. Locally, restart with USE_DFLASH=1 bash serve.sh. For the container, repeat its run command with -e USE_DFLASH=1 before the image name.

The helper uses additional GPU memory, leaving less room for the conversation. Compare it for interactive use; start with plain generation for long documents. Read the fitted context length from the startup log and measure speed on your own card.