Qwen3.8-27B-SSMFIX-apostate-GSQ-RCO-IQ3_S — NInfer artifact

Single-file NInfer engine artifact for the GSQ-RCO IQ3_S mixed-precision quantization of Qwen3.8-27B (SSMFIX-apostate lineage). Built with the GGUF-preserving recipe qwen3_8_27b_gguf: GGUF blocks are imported byte-for-byte (no re-quantization), and everything needed at runtime — text tower, native MTP head, vision tower + merger — lives in one .ninfer file.

Hardware & runtime environment

• GPU: NVIDIA RTX 3080 20 GB (Ampere, sm_86), driver 580.126.20
• CUDA 13.2 (LD_LIBRARY_PATH=/usr/local/cuda-13.2/lib64), container Debian 13 (trixie), 48 GB RAM
• Engine: ninfer fork Ryan-gsq/ninfer-16g-5070ti-5080-5090-qwen3.8-27b-gsq-rco, release line franken-v0.11, built from source at commit e9c5dfe
• The fork's published binaries target sm_120a (Blackwell). On sm_86 boxes build from source instead — the GGUF-preserving converter runs CPU-only, so a source-built engine is all you need.

Upstream patches required on sm_86 (all verified in this deployment)

1. Arch flag: CMAKE_CUDA_ARCHITECTURES=86 (bare 86, this fork rejects 86-real) and -DCMAKE_CXX_SCAN_FOR_MODULES=Off.
2. Apt deps: ninja-build pkg-config libavformat-dev libavcodec-dev libavutil-dev libswscale-dev libcurl4-openssl-dev cuda-nvtx-13-2. On CUDA 13 the nvtx header was split into its own package; missing it fails the build on nvtx3/nvToolsExt.h.
3. Geometry table (src/ops/weight_input.cpp): the QKV input-projection whitelist matches drafter q-shapes by exact equality; the vision merger fc1 pair {4352,1152} (llama.cpp pads the HF 4304 width to multiples of 64) must be registered, otherwise startup aborts with unsupported logical geometry.
4. Converter inventory already registers the ggml block ids used by GSQ-style mixes (iq3_s=21, q3_k=11, q8_0=8, iq4_xs=23, plus the iq2/q2/q4_k/q6_k families); unknown block types fail loudly.
5. One tensor in the source GGUF (blk.64.nextn.eh_proj.weight, Q4_0) is outside the engine's registered block set; it was repacked losslessly to Q8_0 before conversion (same fp16 scales, round-trip maxdiff measured at exactly 0.0). No other byte changed — all remaining tensors import bit-identical.

Rebuild after editing the tables: rm -rf build && cmake -S . -B build -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CXX_SCAN_FOR_MODULES=Off && ninja -C build ninfer-serve -j8 (~10 min). Incremental ninja can hit a ninja dyndep assertion; wipe and rebuild.

Serving parameters (production unit, RTX 3080 20 GB)

ninfer-serve qwen3_8_27b_apostate_iq3s_vision_q8.ninfer \
  --model-id Qwen/Qwen3.8-27B --max-context 262144 --kv-capacity 262144 \
  --max-concurrency 2 --prefill-chunk 1024 --kv-dtype rk4v4 \
  --spec mtp --draft-tokens 2 --lm-head-draft --gdn-state-fp16 \
  --vision-cpu --vision-max-merged 1024 \
  --host-state-slots 8 --host-kv-mib 16384 \
  --fast-prefill-kernel --context-cache-policy rolling \
  --auto-long-anchors --long-anchor-spacing 512 --release-diverged-checkpoints \
  --auto-prefix-grid --kv-lease-growth --concurrent-prefill \
  --disk-kv-path ./ninfer-diskkv --disk-kv-gib 8 --disk-kv-restore \
  --stats-port 8081 --host 0.0.0.0 --port 8080

Notes that matter for reproducing the numbers:

• --model-id Qwen/Qwen3.8-27B — served id in /v1/models; the engine matches the request's model field strictly, so clients must send exactly this string (gateway aliases map to it). Without the flag the id falls back to the artifact metadata name.
• --max-context 262144 --kv-capacity 262144 — per-request ceiling and KV pool, both the full 256 K window. With lease-growth (below) raising max-context costs no extra VRAM: the address table widens while allocated pages stay ≈ frontier window + 4 K. Net effect: weights ~11.3 GiB + runtime ~7.46 GiB, ~1.5 GiB device headroom, no eviction at c2.
• --max-concurrency 2 — two simultaneous lanes; this is what fits on a 20 GB card with the IQ3_S weight band. Three int8-KV lanes measured short (~8.7 vs ~8.3 GiB available).
• --prefill-chunk 1024 — chunked prefill keeps long prompts from starving decode. Chunk 2048 measured only +2 % decode / +1.7 % prefill for ~1.1 GiB extra VRAM, and the cublas fast path needs ≥ 4096 which doesn't fit — stay at 1024.
• --kv-dtype rk4v4 — Hadamard-rotated 4-bit keys and values (rk8v4 = K8/V4). ~31 % smaller KV pages than rk8v4 at equal retrieval fidelity: exact needle recall at ~65K/131K/147K depths, +0.10 % ppl vs int8 KV. On Ampere bandwidth this is what funds the full-depth lanes.
• --spec mtp --draft-tokens 2 --lm-head-draft — native-MTP speculative decoding (head baked into the -mtp source GGUF). --lm-head-draft consumes the packed proposal head and requires the artifact to carry it (--proposal at pack time, else startup fails naming the missing piece). n=2 holds accept rates 54–82 %; n=3 collapses to ~40 % on long Chinese outputs. An ngram copy draft is additionally on by default (n=15) whenever --spec is set — pass --ngram-draft-tokens 0 for a clean MTP-only measurement.
• --gdn-state-fp16 — recurrent states of the gated-linear-attention layers in fp16; matches this fork's storage and the protocol the quality gate was run under.
• --vision-cpu --vision-max-merged 1024 — whole vision path (tower + merger) on CPU threads; each image contributes ≤ 1024 merged tokens to the prompt stream. Keep both together: GPU-resident vision costs ~0.5 GB of KV headroom and buys only tens of ms (see CPU offload design).
• --host-state-slots 8 --host-kv-mib 16384 — host tier: 8 pinned decoder states + 16 GiB pinned KV for dormant sessions. Charged to container RAM, zero GPU cost, gives instant resume after idle.
• --context-cache-policy rolling --auto-long-anchors --long-anchor-spacing 512 --release-diverged-checkpoints — the append-only history cache. Rolling keeps one live checkpoint frontier per session; anchors every 512 tokens let a mid-history rewrite keep everything before the rewrite point (~54 % hit on the first turn after compression, back to ~99.5 % next turn); diverged checkpoints are GC'd lazily. Spacing ≤ 1024 matters — 2048 was too sparse for ~1.5 K-token turns (root-only survival, ~4 %).
• --auto-prefix-grid — fresh sessions sharing an identical message head inherit it across lanes (~53–54 % first-request hit). Hit rates reading 0 right after a restart are warm-up, normal.
• --kv-lease-growth --concurrent-prefill — KV grows per lease (frontier + 4 K window + cushion, not the whole output budget), and idle lanes admit new prefills during staged prefill. Verified: a 3 × 800-token burst completes fully at c2, TTFT 115–152 ms.
• --disk-kv-path DIR --disk-kv-gib 8 --disk-kv-restore — third tier on disk, keyed by artifact hash; near-zero idle cost, pays back under eviction or restart restore.
• --stats-port 8081 --request-log-jsonl FILE --request-log-max-mib 32 — JSON stats + per-request log; the measured numbers above are parsed from these lines, so quote them from one and the same run.
• --host 0.0.0.0 — engine default is 127.0.0.1; LAN callers and the gateway get connect-failed 500s without this flag.
• Env LD_LIBRARY_PATH=/usr/local/cuda-13.2/lib64, NINFER_CUDA_SYNC=spin — CUDA 13.2 runtime path; spin-wait sync instead of blocking cuts idle→first-request TTFT to ~65 ms with no wake penalty, which is what intermittent agent workloads live on.
• After systemctl restart, read capacity/warmup lines only from the newest journal block — old instances of the same unit log in different formats. And when relaunching by hand after a kill, wait until nvidia-smi reads ≈ 1 MiB, or startup dies with cudaMalloc OOM.

CPU offload design

The vision path is fully CPU-resident (--vision-cpu, 16 preprocess workers, 1 GiB media cache): image decode → resize → patchify → tower → merger run on CPU threads, so a vision request adds only its ~124 merged tokens to the GPU prompt stream. Measured on the 3080: GPU utilization ~0 % while the tower runs, VRAM stays constant, per-image media prep 5–31 ms, repeat of the same image is served from cache (TTFT ≈ 50 ms). The text tower + MTP keep the whole GPU working set.

The engine also tiers KV/state off GPU memory: --host-state-slots 8 --host-kv-mib 16384 pins 8 decoder states + 16 GiB of KV in host RAM for instant resume of dormant sessions, and --disk-kv-path/--disk-kv-gib 8 --disk-kv-restore spills idle KV blocks to disk keyed by artifact hash — near-zero cost when idle, pays back after restarts or eviction.

Measured performance (same harness, medians)

Quality baseline of this lineage (full suites, same harness): GSM8K strict 0.9644 / flex 0.9575, IFEval inst_strict 0.6127, CMMLU 0.7112.

Quick start

ninfer-serve qwen3_8_27b_apostate_iq3s_vision_q8.ninfer --model-id Qwen/Qwen3.8-27B \
  --kv-dtype rk4v4 --spec mtp --draft-tokens 2 --lm-head-draft \
  --vision-cpu --host 0.0.0.0 --port 8080

Then use the OpenAI-compatible endpoint (GET /v1/models returns the served id; image inputs go in as image_url parts with detail: "auto" — high is rejected by design since the fixed vision frontend has no rescale profile).

Chat template

The artifact ships the official Qwen template (compiled whitelist matches it by content hash at pack time). For agent/tool-call clients we run froggeric/Qwen-Fixed-Chat-Templates v22.5 instead — it fixes thinking-block stripping, empty-think poisoning, multi-turn role boundaries and KV-cache-invalidating whitespace, verified here across single/multi-turn probes (think-end boundaries and reasoning_content separation both correct; the artifact itself needs no changes, the template is a drop-in).

Two ways to apply it:

# A) runtime override per launch
ninfer-serve ... --chat-template /path/to/chat_template.jinja

# B) bake into the artifact at pack time (hash gets pinned into the artifact)
python -m tools.convert --model <metaDir> ... --resource chat_template.jinja=/path/to/chat_template.jinja

Pull the template with: hf download froggeric/Qwen-Fixed-Chat-Templates chat_template.jinja.

---

中文说明

Qwen3.8-27B（SSMFIX-apostate 血缘）GSQ+RCO IQ3_S 混合精度量化的 NInfer 引擎单文件产物。 用 GGUF-preserving 配方 qwen3_8_27b_gguf 转换：GGUF 量化块逐字节导入、不重新量化；运行时所需的一切——文本塔、原生 MTP 头、视觉塔 + merger——都在同一个 .ninfer 文件里。

硬件与运行环境

• GPU：NVIDIA RTX 3080 20 GB（Ampere，sm_86），驱动 580.126.20
• CUDA 13.2（LD_LIBRARY_PATH=/usr/local/cuda-13.2/lib64），容器 Debian 13（trixie），48 GB 内存
• 引擎：ninfer fork Ryan-gsq/ninfer-16g-5070ti-5080-5090-qwen3.8-27b-gsq-rco，发布线 franken-v0.11，源码 commit e9c5dfe
• 该 fork 发布的成品二进制面向 sm_120a（Blackwell）；sm_86 机器请从源码构建——GGUF-preserving 转换器本身纯 CPU 运行，有源码编译出的引擎就够。

sm_86 上必需的上游 patch（本部署全部实测通过）

1. 架构旗标：CMAKE_CUDA_ARCHITECTURES=86（裸 86，此 fork 拒绝 86-real）并加 -DCMAKE_CXX_SCAN_FOR_MODULES=Off。
2. apt 依赖：ninja-build pkg-config libavformat-dev libavcodec-dev libavutil-dev libswscale-dev libcurl4-openssl-dev cuda-nvtx-13-2。CUDA 13 把 nvtx 头拆成了独立包，缺它会在 nvtx3/nvToolsExt.h 处中断编译。
3. 几何表（src/ops/weight_input.cpp）：QKV 输入投影白名单按精确相等匹配 drafter 的 q 形状；视觉 merger 的 fc1 对 {4352,1152}（llama.cpp 把 HF 的 4304 宽度补齐到 64 的倍数）必须登记，否则启动即报 unsupported logical geometry。
4. 转换器清单已登记 GSQ 风格混合用到的 ggml block id（iq3_s=21、q3_k=11、q8_0=8、iq4_xs=23，另含 iq2/q2/q4_k/q6_k 家族）；遇到未知 block 类型会明确报错。
5. 源 GGUF 里有一个张量（blk.64.nextn.eh_proj.weight，Q4_0）不在引擎登记的 block 集合内；转换前已无损重打包为 Q8_0（同为 fp16 缩放，往返 maxdiff 实测恰好 0.0）。其余字节未动——所有剩余张量逐位导入。

改完表后重建：rm -rf build && cmake -S . -B build -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CXX_SCAN_FOR_MODULES=Off && ninja -C build ninfer-serve -j8（约 10 分钟）。增量 ninja 可能触发 ninja dyndep 断言；清掉重建。

服务参数（生产单元，RTX 3080 20 GB）

ninfer-serve qwen3_8_27b_apostate_iq3s_vision_q8.ninfer \
  --model-id Qwen/Qwen3.8-27B --max-context 262144 --kv-capacity 262144 \
  --max-concurrency 2 --prefill-chunk 1024 --kv-dtype rk4v4 \
  --spec mtp --draft-tokens 2 --lm-head-draft --gdn-state-fp16 \
  --vision-cpu --vision-max-merged 1024 \
  --host-state-slots 8 --host-kv-mib 16384 \
  --fast-prefill-kernel --context-cache-policy rolling \
  --auto-long-anchors --long-anchor-spacing 512 --release-diverged-checkpoints \
  --auto-prefix-grid --kv-lease-growth --concurrent-prefill \
  --disk-kv-path ./ninfer-diskkv --disk-kv-gib 8 --disk-kv-restore \
  --stats-port 8081 --host 0.0.0.0 --port 8080

复现实测数字必须知道的细节：

• --model-id Qwen/Qwen3.8-27B —— /v1/models 里的服务 id；引擎严格匹配请求的 model 字段，客户端必须原样发送这个字符串（网关别名映射到它即可）。不带该旗标时，id 回落到产物元数据里的名字。
• --max-context 262144 --kv-capacity 262144 —— 单请求上限与 KV 池，都取满 256K 窗口。配合下面的 lease-growth，抬高上限不多花显存：加宽的是地址表，实占页数 ≈ 前沿窗口 + 4K。净效果：权重 ~11.3 GiB + runtime ~7.46 GiB，设备余量 ~1.5 GiB，c2 下不驱逐。
• --max-concurrency 2 —— 两条并发车道；这是 IQ3_S 权重档在 20 GB 卡上的容量线。int8 KV 三车道实测差 ~0.4 GiB（需 ~8.7、可用 ~8.3 GiB）装不下。
• --prefill-chunk 1024 —— 分块 prefill，避免长提示饿死 decode。chunk 2048 实测只换来 +2 % decode / +1.7 % prefill，却多花 ~1.1 GiB 显存；而 cublas 快路需要 ≥ 4096，放不下——停在 1024。
• --kv-dtype rk4v4 —— Hadamard 旋转的 4-bit K 与 V（rk8v4 是 K8/V4）。同等检索保真度下 KV 页比 rk8v4 再小 ~31 %：~65K/131K/147K 深度 needle 逐字命中，ppl 相对 int8 KV 只 +0.10 %。在 Ampere 的带宽条件下，这就是撑起全深度车道的来源。
• --spec mtp --draft-tokens 2 --lm-head-draft —— 原生 MTP 投机解码（头烘在 -mtp 源 GGUF 里）。--lm-head-draft 消费打包好的 proposal 头，要求产物带它（pack 时用 --proposal，否则启动会点名报缺失）。n=2 实测接受率 54–82 %；n=3 在长中文输出上塌到 ~40 %。另外，只要设了 --spec，默认还会叠加一个 n=15 的 ngram 拷贝草稿——纯测 MTP 请加 --ngram-draft-tokens 0。
• --gdn-state-fp16 —— 门控线性注意力层的递归状态用 fp16；与本 fork 的存储方式一致，也是跑质量门时采用的协议。
• --vision-cpu --vision-max-merged 1024 —— 视觉全链路（塔 + merger）跑 CPU 线程；每张图向提示流最多贡献 1024 个合并 token。两个旗标配套使用：视觉放 GPU 会吃掉 ~0.5 GB KV 余量，只换回几十毫秒（见 CPU offload 设计）。
• --host-state-slots 8 --host-kv-mib 16384 —— 主机层：8 个钉住的解码器状态 + 16 GiB 钉住的 KV，服务闲置会话。记在容器内存账上、零 GPU 成本，闲置后立即恢复。
• --context-cache-policy rolling --auto-long-anchors --long-anchor-spacing 512 --release-diverged-checkpoints —— 追加式历史缓存。rolling 为每个会话保留一个活跃检查点前沿；每 512 token 打一个锚点，使中间改写能保住改写点之前的全部内容（压缩后首轮命中 ~54 %，下一轮回 ~99.5 %）；分叉的检查点惰性回收。spacing ≤ 1024 很关键——2048 对 ~1.5K token 的轮次太稀疏（只剩根节点存活，命中 ~4 %）。
• --auto-prefix-grid —— 共享同一消息开头的新会话可跨车道继承（首请求命中 ~53–54 %）。刚重启后命中显示 0 属预热，正常。
• --kv-lease-growth --concurrent-prefill —— KV 按租约增长（前沿 + 4K 窗口 + 余量，而不是整个输出预算），且空闲车道在分阶段 prefill 期间允许接纳新 prefill。已验证：c2 下 3 × 800 token 的突发全部完成，TTFT 115–152 ms。
• --disk-kv-path DIR --disk-kv-gib 8 --disk-kv-restore —— 磁盘第三层，按产物哈希寻址；闲置近乎零成本，在驱逐或重启恢复时回本。
• --stats-port 8081 --request-log-jsonl FILE --request-log-max-mib 32 —— JSON 统计 + 逐请求日志；上文实测数字都从这些行解析而来，引用时请取同一次运行。
• --host 0.0.0.0 —— 引擎默认只监听 127.0.0.1；不加这条，局域网调用方和网关会拿到连接失败的 500。
• 环境变量 LD_LIBRARY_PATH=/usr/local/cuda-13.2/lib64、NINFER_CUDA_SYNC=spin —— CUDA 13.2 运行时路径；spin 等待式同步取代阻塞，把闲置→首请求 TTFT 压到 ~65 ms 且无唤醒惩罚，这正是间歇性 agent 负载赖以生存的。
• systemctl restart 之后，只从 journal 最新的一段里读容量/预热行——同一 unit 的旧实例日志格式不同。手工杀进程后重启前，等 nvidia-smi 显示 ≈ 1 MiB，否则启动会因 cudaMalloc OOM 失败。

CPU offload 设计

视觉链路全部驻 CPU（--vision-cpu，16 个预处理 worker，1 GiB 媒体缓存）：图像解码 → 缩放 → patchify → 塔 → merger 全在 CPU 线程上跑，所以一次视觉请求只把它的 ~124 个合并 token 注入 GPU 提示流。3080 上实测：跑塔期间 GPU 利用率 ~0 %、显存恒定，单图准备 5–31 ms，同一张图重问走缓存（TTFT ≈ 50 ms）。文本塔 + MTP 占用整个 GPU 工作集。

引擎还把 KV/状态分层出显存：--host-state-slots 8 --host-kv-mib 16384 在主机内存钉住 8 个解码器状态 + 16 GiB KV，闲置会话可瞬时恢复；--disk-kv-path/--disk-kv-gib 8 --disk-kv-restore 把闲置 KV 块按产物哈希溢写到磁盘——闲置时近乎零成本，重启或驱逐后回本。

实测性能（同 harness，中位值）

本血缘的质量基线（完整套件，同 harness）：GSM8K strict 0.9644 / flex 0.9575、IFEval inst_strict 0.6127、CMMLU 0.7112。

快速开始

ninfer-serve qwen3_8_27b_apostate_iq3s_vision_q8.ninfer --model-id Qwen/Qwen3.8-27B \
  --kv-dtype rk4v4 --spec mtp --draft-tokens 2 --lm-head-draft \
  --vision-cpu --host 0.0.0.0 --port 8080

然后走 OpenAI 兼容接口（GET /v1/models 返回服务 id；图像以 image_url part 传入并使用 detail: "auto"——high 被设计性拒绝，因为固定视觉前端没有重缩放档位）。

Chat template（中文）

产物内嵌官方 Qwen 模板（编译期按内容哈希匹配进白名单）。面向 agent/工具调用客户端，我们改用 froggeric/Qwen-Fixed-Chat-Templates v22.5：修复 thinking 块剥离、空 think 毒化、多轮角色边界与破坏 KV 缓存的空白符问题——本部署单轮/多轮探针实测通过（think-end 边界与 reasoning_content 分离均正确），产物本身无需改动，模板即插即用。

应用方式二选一：

# A) 启动时运行时覆盖
ninfer-serve ... --chat-template /path/to/chat_template.jinja

# B) pack 时烘进产物（哈希固定进 artifact）
python -m tools.convert --model <metaDir> ... --resource chat_template.jinja=/path/to/chat_template.jinja

拉取模板：hf download froggeric/Qwen-Fixed-Chat-Templates chat_template.jinja。