Swift-Bonsai-2 27B — NInfer

Swift-Bonsai-2 is UkisAI's reasoning-efficient derivative of Prism ML's Ternary Bonsai 2 27B, packaged here for the NInfer engine as PQ2_0_G128 and PTQ1_0_G128 artifacts.

The ternary codes were moved byte-for-byte out of the source GGUF into the NInfer container — nothing was dequantized and requantized, so there is no second quantization loss.

Quantizations

Both files are in the root of this repository — download them from the Files tab. Each file is a complete container — text tower, vision tower, MTP head, proposal head, tokenizer, chat template and media-processor resources. No adapter file, patch, or extra flag is needed. No DFlash2 adapter.

Engine compatibility

This artifact uses the PQ2_0_G128 / PTQ1_0_G128 dialect and needs an engine from the Ambolio/ninfer-4090-windows lineage (v1.0.6 / v1.0.8).

Where to get the engine: the original Ambolio repos were taken down by their author (the account shows zero public repos; the HF mirror 404s). The lineage persists through forks — engine source, per-GPU build notes, and the converter toolchain that produced this artifact are at 5258MF/ninfer-bonsai2-converter.

It is not interchangeable with the other Bonsai .ninfer artifacts on the Hub:

To check which dialect a .ninfer uses, read the "format" fields in its first 1 MiB.

Requirements

The engine from the Ambolio lineage, built or released for your GPU. KV dtype is architecture-gated:

Quick start

ninfer-serve.exe bonsai2_27b_swift_pq2.ninfer ^
  --host 127.0.0.1 --port 8087 ^
  --max-context 131072 --kv-capacity 131072 --kv-dtype int8 ^
  --spec mtp --draft-tokens 3

On a 12 GB card, --max-context 32768~131072 with --kv-dtype int8 was the configuration measured below. Scan the MTP window (1–4) on your own card — see Measured.

Measured

RTX 3060 12G, production parameters (ctx 65536 / int8 / MTP d4+lm / vision), greedy, 400 output tokens, 1 warmup + 2 runs (median), both models started alternately.

Perplexity, same corpus as the baseline: 8.073669 / 26.962015 / 148.3656 (Swift) versus 8.079208 / 26.972171 / 148.3863 (base) at window/stride 512/256, 32/16 and 8/4 — Swift is lower by 0.07 %, 0.04 % and 0.01 %. Swift and base stay within ±2 % on throughput in every configuration.

MTP settings: the best draft depth depends on how predictable the content is. On the official bench (single-token free continuation) d4+lm leads: 76.6 vs 71.4 t/s at tg256, 62.0 vs 58.8 at tg128. Over HTTP with real 400-token generations d3 leads instead: 50.3 vs 47.6 t/s (greedy 52.0 vs 51.7). The gap on real content is ~1–5 %, near the noise floor; the bench gap is larger.

A caveat worth knowing: acceptance rate is a property of the content, not of the card. A verbatim-repeat prompt pushes acceptance past 96 % and makes the largest draft look best by construction. Measure with prompts that resemble your own workload.

Full throughput table (four prompt types × four MTP tiers)

Format: t/s (MTP acceptance). Chinese / English / Code / Thinking.

Absolute PPL values are corpus-dependent. Reference figures quoted elsewhere (6.448742 / 26.049634 / 121.157720) come from a different corpus and are not comparable to the numbers above; only the Swift-versus-base delta within one corpus is meaningful.

Scope of validation — what was and was not checked

Validated end-to-end by the publisher on one RTX 3060 12G (2026-09-28): server startup, coherent English and Chinese output, multi-turn prefix reuse (three same-prefix requests, 200/200/200/200), ten consecutive requests, sampling, all four reasoning tiers, vision (a red square, CAT, a blue 42), and tool calling. Prefix reuse measured 2848 ms → 258 ms on a ~2.4k-token prompt.

On the advertised "~40 % fewer thinking tokens": that figure is the upstream GPQA-Diamond result. The same upstream table reports −7.1 % on C-Eval, +0.4 % on IFBench and +5.1 % on AIME 2025 — it is not a general property. Our AIME 2024 test (20 problems × 2 rounds) points the same way: accuracy 37/40 vs 35/40, total output tokens −9 %, but the paired per-problem ratio has a median of 1.01 — for a typical problem the two models reason for the same length. GPQA was not tested here.

The prefix-reuse crash documented for older engine builds did not reproduce here.

Verify

Get-FileHash bonsai2_27b_swift_pq2.ninfer -Algorithm SHA256
# cc54be3800099ada67165ad352450be83d28e8c4d423be9cae573a7b6e6350a0

Get-FileHash bonsai2_27b_swift_ptq1.ninfer -Algorithm SHA256
# cc9e890728ea7357b1d8a0797a4accdc6ca143031471a63e49c9cf6c8314b6ae

SHA256SUMS and artifact-manifest.json carry the same values.

Reproducing it

python -u pack.py build bonsai2_27b_swift_pq2.ninfer \
  --gguf Swift-Bonsai-2-PQ2_0.gguf \
  --template qwen3_8_27b_huihui_abliterated.ninfer

Source and template hashes, the full procedure and the verification methodology are in REPRODUCE.md.

License and attribution

Bonsai 2 27B (original weights)   © Prism ML, Inc.     Apache-2.0
                                 huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf
Qwen3.8-27B (geometry base)       © Alibaba Cloud      Apache-2.0
Swift fine-tune (weight source)   ukisai               Apache-2.0
                                 huggingface.co/ukisai/Swift-Bonsai-2-GGUF
Packer                            pack.py — shensanshu/ninfer-ada-ternary (Apache-2.0)
Container format and engine       github.com/Neroued/ninfer (Apache-2.0)

"Created using Bonsai by Prism ML."

See NOTICE, which also discloses one non-Apache link upstream in the fine-tune family. Apache-2.0 — see LICENSE.

"Qwen" is a trademark of Alibaba Cloud; "Bonsai" and "Prism ML" belong to Prism ML, Inc. This is an unofficial, community-produced derivative and is not endorsed by or affiliated with Alibaba Cloud, Prism ML, ukisai, shensanshu, or the NInfer project.

Intended use and limitations

Research and local inference. Not validated for production, safety-critical, or high-stakes use. All figures above were measured in one hardware/software environment and will differ across GPU, driver, CUDA version and memory bandwidth.

中文说明