GitHub | Engine + Build | Runtime Guide | QSA Paper | X | License

Qwen3.8-27B QSA - BF16 source weights

The original 18 backbone, vision, and MTP shards are byte-for-byte unchanged from Qwen/Qwen3.8-27B. This release adds 52,434,944 FP32 indexer parameters to the 16 full-attention layers. The other 48 decoder layers retain their original Gated DeltaNet architecture. Training left the backbone, vision encoder, projector, and MTP weights unchanged.

The original checkpoint is included as indexers/qsa_indexers.pt. It originates from higa2515/qwen38-27b-qsa-indexers and was trained for 300 steps at length 16,384 with a 50% multimodal mix and synthetic OCR, table, and shape tasks. The training source snapshot is included under research/.

QSA in brief

Qwen Sparse Attention (QSA) uses a learned, compressed indexer to select context in four-token blocks. The 16 full-attention layers in this 27B adaptation use fixed K=2,048 plus the current block. All original KV entries are retained, so this is not KV-cache compression. The 12.5% figure describes K at 16k, not a constant ratio at every context length.

QSA architecture reproduced from Figure 3 (p. 5) of On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability.

Two execution paths

The Transformers wrapper in this repository is a correctness reference. Text uses an SDPA mask, image and video conversations use dense attention, and MTP execution is not implemented. This wrapper does not provide native sparse-kernel speed.

For deployment, use the GGUF release with Qwen3.8-27B QSA for llama.cpp. The engine supports native text, image, and video QSA with M-RoPE, MTP with up to two draft tokens, and prefix-cache reuse. Stock llama.cpp does not support these QSA files. See the runtime guide for build and launch instructions.

Reference loading

Install requirements.txt. The captured Python environment used torch 2.14.1+cu126 and torchvision 0.29.1+cu126; the native runtime separately uses CUDA 12.9. Inspect the packaged custom code before enabling it.

from transformers import AutoModelForImageTextToText, AutoProcessor

path = "higa2515/Qwen3.8-27B-QSA-BF16"
processor = AutoProcessor.from_pretrained(path, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    path, trust_remote_code=True, dtype="auto", device_map="auto",
    attn_implementation="sdpa",
)

Restore the original model

original/ contains the pinned official metadata, including the original config, weight index, processor, tokenizer, license, and model card. The additional QSA shard is excluded by the original index. After downloading this complete repository:

python restore_original.py --source . --output ../Qwen3.8-27B-original

The tool verifies all original LFS hashes and then creates a separate, dense, standard HF snapshot, hard-linking large unchanged weights where possible. It refuses to overwrite an existing output directory.

Limited quality checks

These BF16 research-reference checks compare QSA with the original dense path. They are not Q8_0 native quality results.

The DocVQA probe is not standard open-ended DocVQA scoring. DocVQA was used for evaluation, not as the synthetic indexer training set. These small checks do not establish a general performance-retention percentage or broad quality equivalence.

Native image/video and cache probes passed, with limited multimodal checks through 64k. General multimodal quality at 128k/262k remains unverified.

Measured QSA/Dense throughput

The GGUF engine was measured against its dense Q8_0 control with the same weights, FP16 KV, MTP draft2, engine, and two sm70 32GB GPUs.

All values are token/s. At 261,120 input tokens, QSA was 1.793x faster in decode and 2.921x faster in cold prefill. From each path's re-measured 0k point, mixed decode declined 16.66% for QSA and 54.06% for dense.

These are text-speed results, not long-context reasoning accuracy. QSA's nonzero-depth samples were retained from the earlier matched run, and the two paths were measured in separate sessions. See the full comparison data and methodology.

Credits and licenses

The native engine builds on pentaCoxian/Qwen-V100-llama.cpp. See Contributors and credits for the project roles and upstream notices.

Original model weights and notices retain the upstream Apache-2.0 license. QSA additions in this release are provided under Apache-2.0. Native llama.cpp code retains MIT and its included third-party notices.