GitHub | Engine + Build | Runtime Guide | QSA Paper | X | License

Qwen3.8-27B QSA - Q8_0 GGUF and native runtime

This repository provides a deployable Q8_0 GGUF and captured native runtime. The inference engine is Qwen3.8-27B QSA for llama.cpp, with text, image, and video QSA, M-RoPE, MTP, and live prefix-cache reuse.

The runtime archive includes the complete patched source, so a separate GitHub checkout is not required.

BF16 source weights, the indexer checkpoint, and original-model restoration are published separately. Training left the pretrained backbone, vision encoder, projector, and MTP weights unchanged. All 80 GGUF indexer tensors match the published FP32 indexer export exactly after the documented norm transformation. The remaining 866 tensors are identical to the prior Q8_0 conversion, not to the unquantized BF16 bytes.

QSA in brief

Qwen Sparse Attention (QSA) uses a learned, compressed indexer to select context in four-token blocks. The 16 full-attention layers in this 27B adaptation use fixed K=2,048 plus the current block. The other 48 Gated DeltaNet layers retain their architecture. All KV entries remain allocated: QSA reduces attention computation and KV reads, not KV-cache storage.

QSA architecture reproduced from Figure 3 (p. 5) of On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability.

Run

Stock llama.cpp does not implement this QSA path. Extract the runtime archive next to the two GGUF files and run its launcher:

sha256sum -c SHA256SUMS
tar -xzf qsa27b-linux-x86_64-sm70.tar.gz
QSA_MODEL_DIR="$PWD" ./qsa27b-runtime/launch.sh

The captured binary targets Linux x86_64 and NVIDIA sm70. It was used with two 32GB GPUs (Tesla V100-PCIE and PG500), driver 580.178.04, CUDA toolkit/runtime 12.9, layer split 1:1, FP16 KV, context 262,144, batch 2,048, ubatch 512, and MTP max draft 2. The host NVIDIA driver and CUDA 12 libraries are prerequisites and are not redistributed in the archive. Other hardware needs a source build and appropriate memory sizing. The default address is 127.0.0.1:8091; set QSA_HOST=0.0.0.0 if required. No service is installed automatically.

Prefix reuse covers target KV, pooled indexer keys, and MTP carry in one live slot. It does not persist across server restarts. MTP lag/async are disabled. In the BF16 Transformers reference, text uses masked SDPA QSA, while image and video conversations fall back to dense attention. MTP execution is not implemented in that reference.

Limited quality checks

These BF16 research-reference checks compare QSA with the original dense path. They are not Q8_0 native quality results.

The DocVQA probe is not standard open-ended DocVQA scoring. These small checks do not establish a general performance-retention percentage or broad quality equivalence.

Measured QSA/Dense throughput

The same Q8_0 GGUF, FP16 KV, MTP draft2, engine, and two sm70 32GB GPUs were used for both paths. Dense restores the 16 original full-attention layers with LLAMA_QSA_DENSE=1; the 48 Gated DeltaNet layers stay unchanged. The text workload mixed two coding tasks and two free-prose tasks.

All values are token/s. At 261,120 input tokens, QSA was 1.793x faster in decode and 2.921x faster in cold prefill. From each path's re-measured 0k point, mixed decode declined 16.66% for QSA and 54.06% for dense. The 0k prompts still contain 490-563 tokens.

QSA's nonzero-depth samples were retained from the earlier matched run; QSA and dense were measured in separate sessions, not interleaved A/B repeats. No confidence interval or general quality claim is made. See benchmark/ for the full data and methodology.

Native multimodal probes are limited to checks through 64k. Broad image/video quality, long-document quality, and 128k/262k multimodal quality remain unverified. The BF16 research checks do not establish quality equivalence for Q8_0 native inference.

Credits and licenses

The engine builds on pentaCoxian/Qwen-V100-llama.cpp at 6209237df53569f07429f3f79fc5354de5fe704f. See Contributors and credits for the project roles and upstream notices.

Model files retain Apache-2.0. The engine archive contains MIT code and third-party notices. See the bundled licenses and provenance.