Qwen3.8-27B GSQ-RCO IQ3_S with distillation-recovered scales (GGUF)

Research artifact, not a general-purpose release.

• What was changed: only the fp16 d/dmin scale fields of every quantised block in 399 tensors, retrained by distillation against the BF16 parent; integer codes, format, byte length and offsets identical to the source file.
• Why it exists: to test whether scale-only recovery brings an existing low-bit file inside the fidelity bar without re-quantising it.
• Parent model: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (IQ3_S), from Qwen/Qwen3.8-27B
• Useful for normal inference? yes, wherever the source IQ3_S file was: same loader path, and it tracks the parent closer (0.0450 / 97.80% vs the source's matched row on this card).
• Experiment: Model Surgery and Scale Recovery
• Evidence: scale-recovery evidence (dataset: run record, logs, fidelity and HellaSwag records, code), also copied under evidence/ here
• Runner compatibility: Runner v0.5.3 or later serves the whole file on CUDA (i-quant kernels on the tensor cores, no CPU fallback); on Metal the IQ2/IQ3 tensors are not implemented, so the file runs on the CPU there. Any engine that reads the source file reads this one.

ISTA-DASLab's GSQ-RCO IQ3_S file of Qwen3.8-27B, with every quantised block's fp16 scales retrained end to end against the BF16 parent by knowledge distillation on a half-prose corpus. Same integer codes, same format, same 11,771,546,784 bytes, same loader path as the source file. The only bytes that changed are the d/dmin scale fields.

Runner fidelity against the BF16 parent on 500 held-out prose positions: mean KLD 0.0450, margin-qualified top-1 97.80% — inside the house bar (≤ 0.05, ≥ 97%). The matched-position row for the unmodified source file is below, so the reader can see exactly what recovery changed.

Serve with xyntetik-runner

runner -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf --serve --port 8080

Runner v0.5.3 or later serves every tensor in this file on CUDA, i-quants included, with prefill on the tensor cores; the whole 11.8 GB file is device-resident on a 16 GB card. On Metal the IQ2/IQ3 tensor types are not implemented, so on Apple silicon the file runs on the CPU as a whole. Any engine that reads the source file reads this one: nothing in the header, the tensor list, or the offsets moved.

Quickstart

runner -m Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf -p "The capital of France is" -n 32

What was changed, exactly

• Changed: the fp16 d field of every block in 399 quantised tensors (103,567,360 blocks), plus the fp16 dmin field where the format carries one (Q2_K, Q4_K). Nothing else.
• Unchanged: every integer code, codebook index and sign; every F32, BF16 and IQ1_M tensor; the embedding and output-head scales; all metadata; all tensor offsets. The output has the source's exact byte length.
• Verified after writing: every patched tensor re-decoded with the reference gguf decoder equals the trained weights; every unpatched tensor is byte-identical to the source. Largest change to any single decoded weight: 0.0195.

RECOVERY.json records the source file's SHA-256, the parent revision, the training checkpoint hash, the output hash, the counts above and the measurements below.

Why scales

GSQ optimises the integer grid assignments with a gradient method and RCO allocates a type per tensor under a size budget; neither trains anything through the model. In a fixed GGUF the remaining continuous degree of freedom is the per-block scale, which is linear in the decoded weight for every block format, so it can be trained end to end against the parent's distribution on the student's own forward pass without reimplementing any codec — the second phase of EfficientQAT applied to an existing file. The distillation blend was half Wikitext-103 prose on purpose: prose is where every low-bit variant of this model separates from its parent, and the one published trained quant of it traded prose away.

Training: 300 steps × 4 sequences × 512 tokens (0.6 M tokens), lr 3e-4 cosine, top-64 forward-KL to the parent with the tail as one bucket, only scales trainable, one decoder block resident on a 24 GB slice at a time.

Measured envelope

Two protocols, kept apart. The first is the release bar. The second is the selection metric used during development and is not comparable to the first.

Runner fidelity — scripts/kld-compare-raw.py, against the BF16 parent, the same 500 held-out Wikitext-2 positions for both rows, tie band 0.5 nats:

House bar: KLD ≤ 0.05 and margin-qualified top-1 ≥ 97%. This file: PASS. Against the unmodified source on the same positions, every reported metric moved in the recovered file's favour (KLD 0.0480 → 0.0450, 261 of 500 positions improved), but the paired 95% interval on the KLD difference is [−0.0098, +0.0038] and includes zero at this sample size — so "passes the bar" is the claim, and "better than the source" is a consistent direction with an unresolved magnitude. A larger-sample comparison is being added.

Selection protocol (development only, not a release claim): top-64 forward KL and top-1 agreement against the parent on the student's own forward, 32 mixed instruction examples and 32 Wikitext-2 prose examples of 512 tokens, measured before and after training.

Benchmark annex (added 2026-09-14): HellaSwag validation, fixed random subsample of 1,000 items (seed 20260911), parent and candidate scored through one float32 reference forward, paired. Fidelity and benchmark equivalence are different quantities — see Limits.

Both differences from the parent fall outside a ±0.5-point equivalence margin: the file is inside the fidelity bar and measurably below the parent on this task, the divergence documented in the Muse-Glimmer surgery report. The source-file row (added 2026-09-15) locates that cost: it is the source quantisation's, and scale recovery neither reduced nor added to it — the recovered and source files disagree on 5–7 of 1,000 items, net +0.1 point, interval through zero.

Limits, read before quoting

• Parent-agreement is not a capability benchmark. The fidelity rows measure how closely next-token distributions track the parent on held-out prose.
• "No quality loss" is not claimed and is not supported by this evidence.
• Two fidelity rows, two protocols. The Runner KLD is variant‖reference over the union of both sides' top-20 log-probabilities; the selection KL is teacher‖student over the top-64 with a tail bucket. They are not interchangeable, and neither is the full-vocabulary KL some cards report.
• Metal runs it on the CPU. Runner serves the file on CUDA since v0.5.3, but the IQ2/IQ3 tensor types have no Metal kernels yet, so on Apple silicon the whole file runs on the CPU.
• Selection data. The training corpus, both development sets and the harness corpus were used to build or select this file. Only the benchmark annex touches none of them.
• Thin margin on agreement. 97.80% against a 97% bar on 500 positions.
• Multimodal not evaluated. The parent includes vision; this is the text backbone and text-only measurements cannot speak to it.
• One model, one corpus blend, one shared 24 GB slice.

Reproduce

# 1. Decompose the source file: integer codes frozen, scales trainable (bit-exact by construction).
python build_unit_cache.py --gguf Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --output cache/
# 2. Teacher targets from the BF16 parent on the training corpus.
python teacher_topk.py --corpus evidence/training-corpus.json --output evidence/teacher-train
# 3. Train the scales.
python train_scales.py --gguf Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --cache cache/ \
    --corpus evidence/training-corpus.json --teacher evidence/teacher-train \
    --steps 300 --batch 4 --lr 3e-4 --warmup 25 --output run/
# 4. Write them back into a byte copy of the source, with verification.
python write_scales.py --source Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf --cache cache/ \
    --checkpoint run/checkpoint-00300.pt --output Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf
# 5. Fidelity in the release protocol.
python scripts/kld-compare-raw.py --model-a Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered.gguf \
    --model-b Qwen3.8-27B-BF16-backbone.gguf --runner ./runner \
    --corpus harness-prose-heldout.txt --max-positions 500 --out fidelity.json

Code and the full evidence record (every anchor, every run, the negative results on the smaller IQ3_XXS file): the code/ folder of this repository (the study is not yet on GitHub) at commit 51cead8.

Provenance

• Parent: Qwen/Qwen3.8-27B, revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, Apache-2.0.
• Source quant: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, Qwen3.8-27B-GSQ-RCO-IQ3_S.gguf, SHA-256 64b53b64c7aa39f2…, Apache-2.0. GSQ: arXiv:2604.18556. RCO: arXiv:2605.00649. The hard part of this artifact — the codes — is theirs.
• This file: SHA-256 873df48dd3ecf205…, 11,771,546,784 bytes.
• Training checkpoint: SHA-256 8a97f066829ec798…, step 300.
• Training data: Wikitext-103 (train) and SmolTalk (train shards 1–8), 2,048,000 tokens, hashes in RECOVERY.json. No development, harness or benchmark text.