Qwen3.8-Flash-Next, calibrated Q5 routed experts — Radiance

A checkpoint-specific calibrated quantization of Qwen/Qwen3.8-Flash-Next for the Radiance inference engine.

This repository contains a single self-contained .rad inference container. It is not a Transformers, safetensors, llama.cpp, or vLLM checkpoint. Use a Radiance build with the Qwen3.8-Flash-Next model support and w4nl/w5nl expert kernels.

Artifact

Quantization recipe

• All 48 calibrated trunk layers use GPTQ with three coordinate-descent sweeps.
• Routed expert down projections use non-linear w5nl (5-bit) codebooks, group 64, E4M3 scales, and FWHT128.
• Routed expert gate/up projections use w4nl (4-bit) with the same transform and group structure.
• Ten selected high-energy routed experts remain BF16.
• Attention, gated-delta-net, shared-expert linears, and output head use the calibrated artifact's INT8/Radiance RTN planes.
• The original BF16 PLE/n-gram table is preserved and intended to remain disk-backed.
• The draft-only logical layer 48 retains its existing RTN planes because it was not part of trunk Gram calibration.
• Calibration used 24,768 RADGRAM2 files captured from the exact original BF16 checkpoint. They are available separately at gostar77/qwen3.8-flash-next-radiance-q5-w5dn64-grams.

The recipe and validation report are included in this repository.

Serving with Radiance

A validated host-specific launcher is included as qwen3.8-flash-next-w5dn64-gptq-bf16ple.sh. It assumes the local Radiance build and model paths from the original validation host; adjust its BUILD and MODEL variables after downloading.

The validated profile was:

RADIANCE_DISABLE_VMEM=1 radiance \
  --model ./qwen4exp-w5dn64-bf16ple-gptq.rad \
  --tp 2 --tp-wire exact \
  --placement expert_tiered --ngram-placement disk \
  --host-pool-mib 12288 --gpu-headroom-mib 96 \
  --kv-cache-dtype fp8 --max-num-seqs 2 \
  --max-model-len 262144 --max-num-batched-tokens 2048 \
  --prefix-cache-mib 12288 --checkpoint-slots 32 \
  --num-speculative-tokens 3

RADIANCE_DISABLE_VMEM=1 is a workaround for the ROCm/kernel virtual-memory mapping hang observed on the validation host; it may not be needed on other systems. With fixed KV allocation, two sequences share approximately one full configured-context budget; this is not a guarantee of two simultaneous 262K-token conversations.

Held-out quality and runtime validation

Compared with the uncalibrated Radiance bootstrap on 66,793 full-vocabulary held-out positions:

The actual FP8-KV serving profile passed the same-cache 2% PPL guard and had lower mean KL than the bootstrap and IQ4R baselines. The separate BF16-KV PPL guard narrowly failed at a 2.1126% ratio; this is documented rather than hidden. These are corpus-specific quantization-fidelity measurements, not a claim of universal benchmark superiority.

Runtime smoke tests passed arithmetic, factual, Python-syntax, native-tool, prefix-cache, and synthetic repetitive 260K-context checks. Diverse long-context retrieval quality was not established.

Warm 512-token validation measured 137.2 tokens/s on the server decode metric (132.6 tokens/s including request wall time) on the original dual-Radeon-AI-PRO-R9700 test host.

License and attribution

The model artifact is derived from Qwen3.8-Flash-Next and is distributed under the Qwen Community License 1.0. The complete license is in LICENSE. In particular, review its commercial Model-as-a-Service and AI Work Assistant provisions before deployment.

The included launcher, recipe, and Radiance-related materials also reference software distributed under Apache-2.0; see LICENSE-RADIANCE and NOTICE-RADIANCE. The base-model license remains applicable to the derived model artifact.

For complete validation details, see README-GPTQ-VALIDATION.md.