Groot-65B-A6B — Gyro-M (TQ2_T)

120 tokens/s on a single RTX 3090. 20.2 GB total VRAM. Still 57 t/s with a 3 GB hot tier — an 8 GB card slot. Our own re-encode of Groot-65B-A6B's routed experts to trellis TQ2_T (2.25 bpw), encoded from the official FP8 teacher. First publisher of this combination (2026-11-03).

The number that matters is not the download size

The Groot team's own card told us a 72 GB Blackwell "can't fit" this model. That compares the download to your VRAM — they are different numbers. Here is the actual breakdown:

Vocabulary used on this card: expool = the complete set of quantized routed-expert weights — the only component that scales with bpw. Hot tier = the slice of the expool the engine keeps in VRAM. Cold store = the slice that stays in host RAM/NVMe. Never compare the expool to the total file: the file also carries the lookup table and unchanged non-expert tensors.

The GPU never sees the 35 GB lookup table. Fully-resident VRAM on a 24 GB card: 20.2 GB (15.3 GiB expert cache + ~5 GB dense/KV/context), with 4 GB to spare.

Fit ladder — every rung measured on our 3090 (capping the expert cache to emulate smaller cards)

Real-card confirmation: the same pack runs fully resident on a Tesla V100-32 (~$500 used): 90 t/s engine decode, 12 GB VRAM spare — measured 2026-11-03 on a CUDA 12 build (CUDA 13 dropped Volta). A $500 2017 datacenter card is a 90 t/s Groot server.

There is no speed cliff as the hot tier shrinks, because a cold expert fetch is only ~1.3 MB over PCIe — bandwidth, not residency, is the limit. For workload honesty: on a broad 1.25M-token corpus (IsValorum's independent Groot coverage data, 100% of experts observed), routing is flatter than on focused workloads: a 65% hot tier covers ~84% of activations, 39% covers ~63%. Speeds above already include that streaming cost.

System RAM: 32 GB works. The 35.2 GB lookup table does not need to be resident — measured, after real use only 5.7 GB of it (16%) is ever touched; evicting the whole table from page cache before every request cost zero measurable speed. The cold part lives on NVMe and is read sparsely. A fast NVMe is the real requirement; 16 GB RAM is the practical floor, 32 GB is comfortable. (The 128 GB in some reports is our bench box, not a requirement.)

Same size, different engine — measured contrast

We also benchmarked mradermacher's static Q2_K GGUF (56.3 GB, ~2.6 bpw — nearly the same file size as this pack) on the same RTX 3090 with stock llama.cpp (b11364, --cpu-moe + lookup table pinned to host RAM, because the 21 GB expool + KV does not fit 24 GB):

Upstream llama.cpp gained an expert cache on 2026-10-07 (#29887, --moe-cache-mib), and its eviction policy moved past plain LRU on 2026-11-06 (#31233, ClassLRU — an 8-class recency+frequency hybrid) — real features in the right direction. The distinction that still matters is reactive vs routing-aware: even a smarter eviction policy decides after the router has already missed; Strata's cache predicts from the router before the miss (measured 85.9% hit) and pre-fills from a learned profile. Third-party comparison on a 16 GB card with a 2-bit Flash-Next pack: llama.cpp + 8 GB expert cache 49.4 t/s vs Strata 85.8 t/s, prefill 206 vs 2,320 t/s (local-llm-lab, 2026-10-16). That gap is the engine, not the bits. For a llama.cpp-native path of this model, mradermacher's repo is the right link — this pack needs the Strata engine above (TQ2_T id 144 is not an upstream type).

Is it still smart?

Same model, same harness (boxy-fastcoder-v1, official prompts + asserts, 2026-11-03). The requant at less than half the expert precision costs ~3 points on the coder bench and keeps 120 t/s — while the same-size llama.cpp 2-bit quant rambles and crawls at 4 t/s on plain offload (an 8 GB --moe-cache-mib lifts llama.cpp 2-bit into the 40–50 t/s class; see the engine note above).

Reading the size column honestly: those are whole files (download size), not the streaming number. The expool itself goes ~38 GB (Q4_K_M) → 16.0 GB (TQ2_T) — pool bytes scale linearly with bpw, 4.8/2.25 = 2.1x, exactly as they should. The file shrinks only 1.45x (78.8 → 54.5 GB) because everything outside the experts — lookup table, dense weights, embeddings — is unchanged. Don't divide one column by the other.

• Groot is a reasoning model. Answers arrive only after a thinking trace; stacks with short output budgets or code stop-words truncate it mid-thought and score zero. If your eval said "Groot is dumb", check whether it read reasoning_content and gave the model room to finish — that explains a large share of such reports.
• Honest fine print: Groot's real failure mode is occasional reasoning-loop degeneration (the trace collapses into a repetition spiral and never emits code). 6 of this pack's 8 HumanEval misses are that mode, counted, not hidden.
• Encode quality: 11.35 dB signal-to-error at 2.25 bpw on real expert tensors (healthy for this rate; our production ladder threshold).
• Everything except the routed experts is byte-identical to the official Q4_K_M release — the lookup table, dense weights, and embeddings are copied verbatim.
• Detailed long benches (standard suites, long-context) are running; results will be added to this card and to the boxy.fit leaderboard (row groot-gyro, harness-stamped).

Quickstart (Strata engine)

This GGUF uses the trellis TQ2_T type (id 144) from the Strata/agention kernel family — upstream llama.cpp cannot load it. The filename carries a Q2_0 tag purely so HF's hardware estimator can parse it: Q2_0 is 2.25 bpw, the exact same rate as TQ2_T, so the size estimates are accurate. The tensors themselves are TQ2_T.

# 1. build the serving pack (seconds)
python tools/iq_pack.py --gguf Groot-65B-A6B-GyroM-Q2_0.gguf --compat-bf16 --out pack-groot-gyro
# 2. serve (Strata >= 0.2.8 / engine 0.1.54; CUDA arch 86+)
strata --pack pack-groot-gyro --native Groot-65B-A6B-GyroM-Q2_0.gguf \
       --expert-cache auto --spec 4 --kv k8v4 --max-context 32768

The engine sizes the hot tier to your free VRAM automatically — the 16 GB and 12 GB rows above are what --expert-cache auto produces on those cards.

What this is, exactly

Groot-65B-A6B is a whole-expert-table pruned Flash-Next (256 experts instead of 512, same per-token active path, ~6B active). We halved the stored expool again by re-encoding it with trellis quantization: experts re-encoded from the official FP8 teacher; all other tensors copied byte-for-byte from the official Q4_K_M GGUF. The result: the full 65B expool fits resident on one consumer card, and the model runs faster than it does streamed at 4-bit.

Run it yourself

The harness behind every number on this card is open: harness/fastcoder.py (HumanEval) and fastcoder_mbpp.py (MBPP). pip install datasets, point it at any OpenAI-compatible endpoint (python fastcoder.py <port> <model>), and you get the same prompts, same asserts, same RESULT line. Ran it on a card we don't have? Submit the RESULT line at boxy.fit — accepted claims go on the leaderboard with your credit.

Attribution & license

• Base model: GrootTeam/Groot-65B-A6B-261023 — thank you to the Groot team.
• llama.cpp quants of this model: mradermacher. Independent expert-coverage/imatrix data: IsValorum.
• Architecture and weights lineage: Qwen/Qwen3.8-Flash-Next.
• License: qwen-community-1.0 (text). This requant is a derivative work distributed under the same terms.
• Quantization tooling: Strata engine trellis kernels (Niko1221/Strata). Benchmarks: boxy.fit.