Bonsai-2-27B DFlash2 fast-fork — 740 tok/s from a 27B dense model on one RTX 5080
This is a build-and-benchmark package, not a new checkpoint: four small patches
for the PrismML llama.cpp fork (branch prism,
prism-v7 snapshot), a tuned speculative-decoding server profile, a benchmark
harness, and the deployment kit that keeps it running 24/7 under WSL2.
Headline, measured live on the deployment (RTX 5080 16GB, WSL2, CUDA graphs on):
| Metric | Value |
|---|---|
Decode, warm steady state (canonical code-v1, hybrid) |
894.9 tok/s engine, 99.9% draft acceptance, 768 tokens in 1.03 s |
Decode, warm (canonical reasoning-v1, pure n-gram) |
854.5 tok/s, 100% acceptance (750/750) — Verified |
Decode, warm (canonical code-v1, pure n-gram) |
625.9 tok/s @ 95.7% — Verified |
Decode, fresh state (canonical code-v1) |
142 tok/s @ 60.6% acceptance |
Decode, fresh state (canonical reasoning-v1) |
146.4 tok/s @ 61.9% acceptance |
DFlash2 only, no n-gram layer (canonical code-v1) |
151.2 tok/s @ 60.6% acceptance |
| TTFT | 172 ms warm / 805 ms fresh (code-v1, 201-token prompt) |
| Prefill | 794 tok/s hybrid steady / 1381 tok/s pure n-gram |
| Peak VRAM | 13.7 GB hybrid / 11.3 GB pure n-gram (ctx 32768) |
| GPU power | 165–240 W during generation (peak 268 W) |
Leaderboard standing (localmaxxing.com, RTX 5080): #1 and #2 overall (894.9 hybrid / 854.5 pure n-gram — the latter Verified), and five of our seven runs are Verified, including every speed class: warm fast (854.5 / 625.9), fresh-state (146.4 / 142.0), and the DFlash2 isolation (151.2). The fastest verified 20B+ results on this GPU by a wide margin. The next-best 20B+ model on this GPU is gpt-oss-20b (MoE) at 222 tok/s; the fastest sub-2B models reach ~450 tok/s. Fresh-state and warm-state runs are both submitted and labeled — see "Methodology notes" for what each state means.
The three layers of speed
- The quant.
PQ2_0(group-128, 2.13 bpw ternary) from dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF — uncensored, 6.8 GB of weights/token traffic, ~6.8 GB/s per token at 740 tok/s sits at ~92% of the RTX 5080's measured DRAM ceiling. Decode is bandwidth-bound; the stack is already at the wall. - The drafting. DFlash2 self-speculative decoding with a
Q4_K draft head (
Bonsai-2-27B-DFlash2-Q4_K.gguf, 1.1 GB, included indraft/) plus the fork'sngram-modlayer with a 96-token window. DFlash2 drafts are cheap (same family, tiny head); n-gram drafting is nearly free and goes from ~58% acceptance on a fresh prompt to 90.9% when the workload revisits content — which real agentic/code-edit sessions constantly do. - The patches. Round 10: +11% fresh decode and a much snappier server, see below.
The patches (patches/, unified diff vs the pristine prism-v7 tarball)
| # | File | Lines | What it does |
|---|---|---|---|
| 0001 | src/llama-model.cpp |
24 | Input-layer GPU offload — the token-embedding row lookup moves off the CPU per-token path; single CUDA split, no host round-trip per token. |
| 0002 | ggml/src/ggml-backend.cpp |
20 | Async graph-input staging — CUDA-graph input buffers are staged on the compute stream instead of synchronously on the caller. Removes a per-token sync stall. |
| 0003 | ggml/src/ggml-cuda/mmvq.cu |
377 | Optional PQ2_0 SoA matvec kernel (Round 11) — structure-of-arrays layout with bias fusion, proven bitwise-identical to the AoS kernel. Default off (no aggregate win at this model's shapes); enable with GGML_CUDA_PQ2_SOA=1. |
| 0004 | ggml/src/ggml-cuda/ggml-cuda.cu |
49 | Remaining Round-10 CUDA-side diff (dispatcher plumbing for the above). |
Applied on top of the PrismML fork with CUDA graphs enabled
(GGML_CUDA_GRAPHS=ON is load-bearing: +70% vanilla decode on this stack).
Reproduce
# base: PrismML llama.cpp fork, branch `prism` (prism-v7 snapshot)
# apply patches/000*.patch with: patch -p1 < patches/0001-....patch
cmake -B build-graphs -DGGML_CUDA=ON -DGGML_CUDA_GRAPHS=ON \
-DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release
cmake --build build-graphs --config Release -j
# serve (see scripts/serve_bonsai.sh for the full flag set)
./build-graphs/bin/llama-server \
-m Bonsai-2-27B-PQ2_0-CRACK.gguf \
--model-draft Bonsai-2-27B-DFlash2-Q4_K.gguf \
--spec-type draft-dflash,ngram-mod \
--spec-draft-n-max 5 --spec-ngram-mod-n-max 96 \
-ngl 999 -ngld 999 -fa on -c 32768 -np 1 -t 16 -tb 32 -b 2048 -ub 512 \
--host 0.0.0.0 --port 8001
Benchmark: scripts/bench_lm.py <probe.json> <n_runs> — a streaming SSE consumer
that reports engine tok/s (server-side timings), wall time, TTFT, prefill rate,
acceptance, and samples GPU power/VRAM in the background. scripts/bench_prefill2.py
measures clean prefill (first long prompt pays graph capture; the second distinct
prompt is the real number). Probe: scripts/bonsai_code600.json (600-token code
generation, temp 0, thinking off).
Methodology notes (read before comparing numbers):
- "Warm steady state" = the same generation task re-run on a server whose n-gram map has seen the content (runs 3–6 of a repeat series). This is the real steady state of any agentic loop that revisits a file or conversation.
- "Fresh state" = first run after server start, cold n-gram map: 133 tok/s.
- All numbers temp 0,
enable_thinking: false, single slot, no other GPU load. - Engine tok/s is the server's own
predicted_per_second; end-to-end HTTP tok/s is ~5–15% lower (streaming + framing overhead).
Deployment kit (deployment/)
Runs 24/7 on WSL2: bonsai.service (systemd, auto-restart), a Windows
session-holder scheduled task (bonsai_keepalive_task.xml) that defeats WSL2's
~20 s idle VM shutdown (this WSL build ignores vmIdleTimeout), and
start_bonsai.cmd / stop_bonsai.cmd helpers. The service stages
serve_bonsai.sh through tr -d '\r' on every start so CRLF can never poison it.
Verification
- 0001+0002+0004: deterministic outputs byte-identical to the prebuilt fork binary on prose; one known greedy near-tie flip on the code probe (float non-associativity in GPU dequant rounding flips a last-bit tie — documented, not a correctness bug).
- 0003:
scripts/parity5.sh/parity6.shprove bitwise parity of the SoA kernel vs the AoS kernel across on/off/default configs under the full production config (draft + graphs). - nsys per-kernel profiling of the whole matvec stack: ~874 GB/s average = 92% of the measured 949 GB/s DRAM ceiling. The kernel lever is exhausted; remaining per-token cost is ~2.4 ms of non-matvec ops (FWHT, norms, get_rows) and ~0.9 ms inter-graph gap.
Contents
patches/ 4 unified diffs vs pristine prism-v7
scripts/ serve profile, bench harness, probes, parity + A/B scripts
deployment/ systemd unit, WSL keepalive task XML, Windows start/stop
results/ raw benchmark JSON + summary
draft/ Bonsai-2-27B-DFlash2-Q4_K.gguf (1.1 GB draft head)
Credits & links
- Base model: dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF (Apache-2.0)
- Draft head lineage: ProCreations/Ternary-Bonsai-2-27B-DFlash2
- Engine: PrismML llama.cpp fork (branch
prism, DFlash2 + PQ formats) - Benchmarks submitted to localmaxxing.com
License: Apache-2.0 (matches the base model and the fork).