Qwen3.8-27B GSQ-RCO IQ3_S with distillation-recovered scales (GGUF)

Research artifact, not a general-purpose release. Read the first bullet before using it.

• Limit first: this file reads 0.41 mean KLD against Qwen's float32 reference, 8x the fidelity bar. It is a learned derivative of Qwen3.8-27B, not a faithful quantization of it. Against Qwen/Qwen3.8-27B at revision 1d4bf0f2 in float32 it reads mean KLD 0.4119 and margin-qualified top-1 85.04% in the house window, and 0.4084 / 86.12% at positions 4,000-5,999 (bar: KLD at most 0.05, margin-qualified top-1 at least 97%). It predicts the reference corpus better than its parent does (perplexity 21.54 against 73.29), so its next-token distribution is a different one, not a noisier copy of the parent's. ISTA-DASLab's unmodified source file reads 0.5616 / 82.48% and 0.4718 / 86.36% on the same positions (perplexity 53.48): the method fits codes and scales on calibration data, and this file's scale recovery moved it toward the parent (Wilcoxon on KLD p 1.7e-10 and 0.0017).
• Earlier anchor, kept as what it measured: against the project's BF16 GGUF of the same parent, on 500 short held-out Wikitext-2 prefixes with a 0.5-nat tie band, mean KLD 0.0450 and margin-qualified top-1 97.80% (plain top-1 87.20%). That number holds for those positions only. It does not describe the file on longer or other text, and it is not the reference reading above.
• What was changed: only the fp16 d/dmin scale fields of every quantised block in 399 tensors, retrained by distillation against the BF16 parent. Integer codes, format, byte length and offsets are identical to the source file.
• Parent model: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (IQ3_S), from Qwen/Qwen3.8-27B at revision 1d4bf0f2 (the parent's only weight set; the file's unquantized tensors match it bit for bit).
• Useful for normal inference? As a small IQ3_S derivative of Qwen3.8-27B, with the limit above: it serves wherever the source file does and reads closer to the parent than the source, but it is not a stand-in for the parent's distribution.
• Experiment: Model Surgery and Scale Recovery
• Evidence: scale-recovery evidence (dataset: run record, logs, fidelity and HellaSwag records, code), also copied under evidence/ here
• Runner compatibility: Runner v0.5.3 or later serves the whole file on CUDA (i-quant kernels on the tensor cores, no CPU fallback); Runner v0.5.6 or later serves it on Metal too (codebook i-quant kernels, byte-exact against the CPU on the fixture gate). Any engine that reads the source file reads this one.
• MTP draft head: Qwen3.8-27B-MTP-Q8_0.gguf in this repo, a speculative-decoding companion (llama.cpp on a GPU: about 1.9x decode; Runner 1.1.3 on CUDA: about 1.1x, 1.5x on its opt-in int8 route; same text); see the head section.

• Validation status (Runner support tiers): each file's row in the Runner roster says which tier it is in, what is re-checked on a release and the date and version of the last run.
  • Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf: roster row
  • Qwen3.8-27B-MTP-Q8_0.gguf: roster row

Want this for another model? The tools behind these files are open: requantization, expert pruning and sublayer removal in Xyntetik Runner, and the method for this file is on this card or in its evidence dataset.

Request a model · Report your result · Community results · All files

If this file is useful to you, a contribution funds the hardware time behind the next one.

ISTA-DASLab's GSQ-RCO IQ3_S file of Qwen3.8-27B, with every quantised block's fp16 scales retrained end to end against the BF16 parent by knowledge distillation on a half-prose corpus. Same integer codes, same format, same 11,771,546,784 bytes, same loader path as the source file. The only bytes that changed are the d/dmin scale fields.

Against Qwen's float32 reference it is outside the fidelity bar: mean KLD 0.41 (8x the bar) and margin-qualified top-1 85-86% in both windows, with a corpus perplexity of 21.54 against the parent's 73.29. GSQ-RCO fits codes and scales on calibration data. The unmodified source reads 0.47-0.56 and is further from the parent; the scale recovery moved the file toward it. The earlier anchor (0.0450 / 97.80% against our BF16 GGUF on 500 short Wikitext-2 prefixes) is in the envelope below, under the reference rows.

The MTP draft head (added 2026-10-08)

Qwen3.8-27B-MTP-Q8_0.gguf is a speculative-decoding draft file for this trunk: with llama.cpp on a GPU that holds trunk and head it decodes about 1.9x faster with the same greedy text, and it is not a model of its own (it cannot generate without the trunk, it does not change what the trunk says, and nothing in it was trained here).

The file (3,164,006,784 bytes, sha256 89e31f13400bac27b888f10bab02ff4d8da1f8610fa078dc32f9f3f0d2b6f6c9) holds the publisher's own multi-token-prediction block: the 15 mtp.* tensors of Qwen/Qwen3.8-27B (revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, shard model-00018-of-00018), exported with llama.cpp's converter (convert_hf_to_gguf.py --mtp --outtype q8_0): norms F32, matrices Q8_0, with the token embedding, output norm and LM head of the same checkpoint that the converter writes beside the block. No other change.

Measured on one 24 GB MIG slice of a datacenter GPU (trunk and head fully on the device), 512-token and 4,096-token prompts, 128 greedy tokens:

The head pays most where the engine's batched rows are nearly free, which is llama.cpp on a GPU. On Runner 1.1.3 it pays a little on CUDA (about 1.1x at --draft-k 2; the default of 4 is slower than plain decoding on this file) and is a correctness feature on the CPU: the drafted text is the plain text, byte for byte. Runner's plain decode on these i-quant formats is still about half of llama.cpp's on the same slice; that is an engine item being worked on, not a property of the head.

# llama.cpp b11433 or newer
llama-server -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf \
  -md Qwen3.8-27B-MTP-Q8_0.gguf --spec-type draft-mtp -ngl 99
# Xyntetik Runner 1.1.3 or newer (CUDA and CPU; 1.1.1 and 1.1.2 draft on the CPU only)
runner -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf --mtp-file Qwen3.8-27B-MTP-Q8_0.gguf --draft-k 2

The head is Qwen's weights under the same Apache-2.0 license as the trunk; the conversion is deterministic from the stated revision.

Serve with xyntetik-runner

runner -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf --serve --port 8080

Runner v0.5.3 or later serves every tensor in this file on CUDA, i-quants included, with prefill on the tensor cores; the whole 11.8 GB file is device-resident on a 16 GB card. Runner v0.5.6 or later serves it on Metal as well: the seven codebook i-quants have Metal kernels, measured 2.2 to 2.5x the CPU and gated byte-exact against the CPU encoders on the 29-fixture set, so on Apple silicon with enough unified memory the file runs on the GPU. Any engine that reads the source file reads this one: nothing in the header, the tensor list, or the offsets moved.

Quickstart

runner -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf -p "The capital of France is" -n 32

What was changed, exactly

• Changed: the fp16 d field of every block in 399 quantised tensors (103,567,360 blocks), plus the fp16 dmin field where the format carries one (Q2_K, Q4_K). Nothing else.
• Unchanged: every integer code, codebook index and sign; every F32, BF16 and IQ1_M tensor; the embedding and output-head scales; all metadata; all tensor offsets. The output has the source's exact byte length.
• Verified after writing: every patched tensor re-decoded with the reference gguf decoder equals the trained weights; every unpatched tensor is byte-identical to the source. Largest change to any single decoded weight: 0.0195.

RECOVERY.json records the source file's SHA-256, the parent revision, the training checkpoint hash, the output hash, the counts above and the measurements below.

Why scales

GSQ optimises the integer grid assignments with a gradient method and RCO allocates a type per tensor under a size budget; neither trains anything through the model. In a fixed GGUF the remaining continuous degree of freedom is the per-block scale, which is linear in the decoded weight for every block format, so it can be trained end to end against the parent's distribution on the student's own forward pass without reimplementing any codec; the second phase of EfficientQAT applied to an existing file. The distillation blend was half Wikitext-103 prose on purpose: prose is where every low-bit variant of this model separates from its parent, and the one published trained quant of it traded prose away.

Training: 300 steps × 4 sequences × 512 tokens (0.6 M tokens), lr 3e-4 cosine, top-64 forward-KL to the parent with the tail as one bucket, only scales trainable, one decoder block resident on a 24 GB slice at a time.

Measured envelope

Three protocols, kept apart. The first is the reference read and the file's fidelity claim. The second is the earlier anchor. The third is the selection metric used during development and is not comparable to the others.

Against Qwen's float32 reference (Qwen/Qwen3.8-27B@1d4bf0f2, measured 2026-11-04; house window / positions 4,000-5,999):

House bar: KLD at most 0.05 and margin-qualified top-1 at least 97%. This file: outside the bar, 8x on KLD. It is closer to the reference than the source in both windows (Wilcoxon on per-position KLD, p 1.7e-10 and 0.0017). A perplexity below the parent's means the file predicts this corpus with a different distribution from the parent's. It does not mean the file is better than the parent.

Earlier anchor, kept as what it measured (500 short held-out Wikitext-2 prefixes, against the project's BF16 GGUF, tie band 0.5 nats):

Runner fidelity, scripts/kld-compare-raw.py, against the BF16 parent, the same 500 held-out Wikitext-2 positions for both rows, tie band 0.5 nats:

On these 500 short prefixes the file read inside the bar. That reading does not carry to the reference rows above, and the reference rows are the file's fidelity claim. Against the unmodified source on the same positions, every reported metric moved in the recovered file's favour (KLD 0.0480 → 0.0450, 261 of 500 positions improved), but the paired 95% interval on the KLD difference is [−0.0098, +0.0038] and includes zero at this sample size, so "passes the bar" is the claim, and "better than the source" is a consistent direction with an unresolved magnitude. A larger-sample comparison is being added.

Selection protocol (development only, not a release claim): top-64 forward KL and top-1 agreement against the parent on the student's own forward, 32 mixed instruction examples and 32 Wikitext-2 prose examples of 512 tokens, measured before and after training.

Benchmark annex (added 2026-09-14): HellaSwag validation, fixed random subsample of 1,000 items (seed 20260911), parent and candidate scored through one float32 reference forward, paired. Fidelity and benchmark equivalence are different quantities; see Limits.

Both differences from the parent fall outside a ±0.5-point equivalence margin: the file is inside the fidelity bar and measurably below the parent on this task, the divergence documented in the Muse-Glimmer surgery report. The source-file row (added 2026-09-15) locates that cost: it is the source quantisation's, and scale recovery neither reduced nor added to it; the recovered and source files disagree on 5–7 of 1,000 items, net +0.1 point, interval through zero.

Limits, read before quoting

• Outside the fidelity bar against the parent's float32 reference (0.41 mean KLD, 8x the bar; margin-qualified top-1 85-86%), and below the parent's perplexity on that corpus: a learned derivative, not a faithful quantization. Quote the reference rows, never the 500-prefix anchor alone.
• Parent-agreement is not a capability benchmark. The fidelity rows measure how closely next-token distributions track the parent on held-out prose.
• "No quality loss" is not claimed and is not supported by this evidence.
• Two fidelity rows, two protocols. The Runner KLD is variant‖reference over the union of both sides' top-20 log-probabilities; the selection KL is teacher‖student over the top-64 with a tail bucket. They are not interchangeable, and neither is the full-vocabulary KL some cards report.
• Metal needs Runner v0.5.6. Runner serves the file on CUDA since v0.5.3 and on Metal since v0.5.6, when the codebook i-quant kernels landed; an older Runner on Apple silicon runs the whole file on the CPU.
• Selection data. The training corpus, both development sets and the harness corpus were used to build or select this file. Only the benchmark annex touches none of them.
• The 97.80% is a short-prefix, margin-qualified reading against our BF16 GGUF (plain top-1 87.20%). It was inside a 97% bar on 500 positions and is superseded by the reference rows.
• Multimodal not evaluated. The parent includes vision; this is the text backbone and text-only measurements cannot speak to it.
• One model, one corpus blend, one shared 24 GB slice.

Reproduce

# 1. Decompose the source file: integer codes frozen, scales trainable (bit-exact by construction).
python build_unit_cache.py --gguf Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --output cache/
# 2. Teacher targets from the BF16 parent on the training corpus.
python teacher_topk.py --corpus evidence/training-corpus.json --output evidence/teacher-train
# 3. Train the scales.
python train_scales.py --gguf Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --cache cache/ \
    --corpus evidence/training-corpus.json --teacher evidence/teacher-train \
    --steps 300 --batch 4 --lr 3e-4 --warmup 25 --output run/
# 4. Write them back into a byte copy of the source, with verification.
python write_scales.py --source Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --cache cache/ \
    --checkpoint run/checkpoint-00300.pt --output Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf
# 5. Fidelity in the release protocol.
python scripts/kld-compare-raw.py --model-a Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf \
    --model-b Qwen3.8-27B-BF16-backbone.gguf --runner ./runner \
    --corpus harness-prose-heldout.txt --max-positions 500 --out fidelity.json

Code and the full evidence record (every anchor, every run, the negative results on the smaller IQ3_XXS file): the code/ folder of this repository (the study is not yet on GitHub) at commit 51cead8.

Provenance

• Parent: Qwen/Qwen3.8-27B, revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, Apache-2.0.
• Source quant: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf, SHA-256 64b53b64c7aa39f2…, Apache-2.0. GSQ: arXiv:2604.18556. RCO: arXiv:2605.00649. The hard part of this artifact, the codes, is theirs.
• This file: SHA-256 873df48dd3ecf205…, 11,771,546,784 bytes.
• Training checkpoint: SHA-256 8a97f066829ec798…, step 300.
• Training data: Wikitext-103 (train) and SmolTalk (train shards 1–8), 2,048,000 tokens, hashes in RECOVERY.json. No development, harness or benchmark text.

About Xyntetik

This repository is published by Xyntetik, the Zenova AB program for running capable language models on hardware you own and proving what they did. Its released engine is Xyntetik Runner: a free, single-binary GGUF engine in C for CPU, CUDA and Metal that serves, scores and trains through the quantized file it deploys and records every run as a receipt. The measurements behind the program's claims are collected at xyntetik.com/evidence. The program's organization on Hugging Face is Xyntetik.