Strata-Sage-A1.5-GGUF

InternScience/Sage-A1.5 (a Qwen3.6-35B-A3B fine-tune, 256 experts / top-8, ~3B active, MTP layer included) as a Q4_K_M GGUF for Strata, running natively on the optimized Qwen3.6 engine path in the q36-r2 branch. It fits the full 262K context and the whole model on a 16 GB card + 64 GB RAM.

The 60 small DeltaNet gate tensors (ssm_alpha / ssm_beta, 2.2 MB) carry Sage-A1.5's original BF16 weights, the form Strata's engine reads; every other tensor and all metadata are byte-identical to the official Sage-A1.5-Q4_K_M-GGUF. It also runs in llama.cpp as any qwen35moe GGUF.

What the branch adds

q36-r2 is upstream Strata 0.2.3 plus two rounds of work on the Qwen3.6 engine path. The files did not change between rounds; only the engine did.

Round 1 (q36-optimized, now proposed upstream with round 2 as #3699 and #3903-#3910): opt-in flags, set in the config below.

Round 2 (this update): on by default wherever the card supports it, so the config needs no new flags. Each one has a --no-<flag> escape. With all of them off, the engine is bit-identical to round 1. Measured on the RTX 5080 below.

Opt-in only: --draft-vocab 16384 --draft-ctx 4096 (MTP drafts over the common and in-context tokens: +7-9% MTP on English and code, slower on other scripts) and --random-seed (a fresh seed for unseeded sampled requests).

Running it

git clone -b q36-r2 https://github.com/xyehya/Strata && cd Strata
./setup.sh          # builds the engine (Linux; Windows: START-HERE.bat)
hf download ykarout/Strata-Sage-A1.5-GGUF --local-dir models/sage
python tools/strata_tokenizer.py --gguf models/sage/Sage-A1.5-Q4_K_M-ssmbf16.gguf --out models/sage/tokenizer

strata-sage.json (the importer only takes Flash-Next files, so this config is written by hand; it is the same as round 1's, because round 2 is on by default):

{
 "exe": "engine/strata",
 "args": ["--native", "models/sage/Sage-A1.5-Q4_K_M-ssmbf16.gguf", "--max-context", "262144", "--prefill", "auto",
          "--kv", "int8", "--pool-workers", "16", "--gqa-attn", "--hybrid-zero-copy", "--expert-prefetch", "2",
          "--expert-profile", "models/sage/expert-profile-agent-v2.prof", "--decode-swaps", "64", "--snapshots", "2",
          "--mtp", "2", "--spec-min-p", "0.5", "--mtp-sampled", "q"],
 "tokenizer": "models/sage/tokenizer",
 "model_name": "sage-a1.5",
 "arch": "qwen35moe",
 "splice_turns": "parsed",
 "sampling": {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5}
}

.venv/bin/python serve/server.py --engine strata --config strata-sage.json --port 8090

OpenAI- and Anthropic-compatible on http://127.0.0.1:8090/v1. "sampling" is InternScience's recommendation (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5) as the server's default for requests that send none: many agent clients (opencode, for one) send no temperature, which otherwise means greedy decoding. A request's own values always win. --pool-workers is the CPU threads for the experts not in VRAM (16 on a 24-core CPU). The profile was learned with 16 GB of VRAM; with less, the engine says so and starts from its default order. The engine prints which round-2 defaults are on and, for any that are off, why (for example "attn-mma off: sm_75").

Measured

RTX 5080 16 GB, Core Ultra 9 285K, 64 GB DDR5. Through the server: fresh code prompts, 384 tokens generated, temperature 0.7, thinking off, MTP 2 as configured, average of two runs per side, run back to back (round 1, round 2, round 1, round 2), tok/s. llama.cpp b11834 tuned for this card (round 1's measurement): --cpu-moe --moe-cache-mib 10300, -b 4096 -ub 2048, MTP draft 2, 8 threads on the P-cores.

The gain grows with context: at 240K, round 2 decodes faster than round 1 did at 40K (162 vs 151 tok/s) and reads prompts 3.1x faster. An agent's next turn on a cached conversation (server, prefix reuse, MTP off): +2,000 tokens on a reused 198K context in 0.96 s (round 1: 5.1 s), +300 tokens in 0.32 s (0.95 s); a cold 198K read takes 67 s (180 s). MTP acceptance is the same as round 1's at every depth (65-73%).

Engine only, no server, forced text, MTP off (the clean kernel number): decode 193 -> 249 (+29%), 170 -> 219 (+29%), 183 -> 229 (+25%) at 4K / 16K / 40K; prompt read +38% / +35% / +42%.

Real agent workload (opencode --auto, one coding task of 52 requests up to 182K context, decode per request grouped by the context it ran at, tok/s). Measured on round 1 only; round 2 has not had an agent run yet:

Close to round 1's sweep row above at each depth (Sage's MTP acceptance is about the same on agent work as on prose), so round 2's sweep gains should carry over. The base model, Qwen3.6-35B-A3B (Strata-Qwen3.6-35B-A3B), accepts 88-95% of its drafts on the same task and decodes 273 / 254 / 234 / 206 tok/s on round 2 in this layout.

Quality. Round 2 changes numerics, so it was checked against a reference rather than against round 1 alone. With the fp32 cache (no KV quantization) as the reference, every round-2 path is at least as close as round 1: mean KL 6.6-8.5e-3 against round 1's 1.55e-2 (full-vocabulary logits, 195 teacher-forced steps at 14K / 57K / 76K), and top-1 agreement 184-189 of 195 against 185. At 200K (65 forced steps), round 2 against round 1 (mean KL 6.8e-3, top-1 63/65) moves the output no more than changing round 1's own prompt chunk size does (6.9e-3, 63/65). Needle retrieval (10 planted codes at 5-95% depth, ~100K and ~200K, with and without MTP): 40/40, as round 1.

License

Apache-2.0, as the original model. All credit for the model to InternScience / Shanghai AI Laboratory (arXiv 2610.28578); this repository only re-packages their weights.