Qwen3.8-27B KVA-Projector (LLKVApprox)

Late-layer KV/state approximation ("KVA-Projector") for Qwen/Qwen3.8-27B — the hybrid (48 GatedDeltaNet + 16 gated full-attention) decoder. Ports the mechanism of kishida's LLKVApprox Qwen3-8B demo (DeepSeek-V4.1-Flash's "Encoder-Decoder" prefill) to the hybrid architecture.

During prefill, run only layers 0..31 for all prompt tokens; this projector fills the approximated layers' cache/state from the boundary hidden state; layers 32..63 then run exactly for a thin slice of the prompt tail (recommended: the last 256 tokens = "exact suffix"). Decode runs all layers exactly. Measured on 1× CMP 170HX (180 W, HF bf16): prefill speedup 1.71× @6,603 tokens with this projector's weights untrained (own-weights init) at suffix 256, and 2.01× with oracle fills — against a 1,983 tok/s full-prefill baseline.

Files

• kva_projector_qwen38_27b.safetensors — projector weights (224 tensors; frozen own-weights W0 bf16 + fp32 trainable: per-layer RMSNorm weight, low-rank delta At(Bt z) rank 128, bias, GELU MLP head hidden 512).
• kva-projector-qwen38-27b.config.json — architecture/training/serving metadata.
• engine/ — the reference LLKVApprox engine for Qwen3.8-27B: engine.py (LLKVEngine: split-32 CED prefill, cache/state plumbing for the hybrid, decode), projector.py (loads the checkpoint), objectives.py, bench_prefill.py + quality_eval.py (reproduction harnesses). Requires: torch 2.13+, transformers (qwen3_5), the BF16 base model, and the projector checkpoint above.

Replicating the 1.71x

1. pip install torch transformers safetensors (qwen3_5-capable transformers build)
2. Download Qwen/Qwen3.8-27B (BF16) + this repo's weights
3. python engine/bench_prefill.py --modes full student --projector <ckpt> --suffix 256 on a >=48GB GPU — measured 3,396 vs 1,983 tok/s (1.71x) on a CMP 170HX @6,603-token prompts; speedup is length-dependent (2.01x oracle ceiling, ~1.1-1.5x below 1k tokens).

Architecture

One head per approximated layer, mapping the boundary hidden state h_enc [T,5120] (rows 0..T-2; the real layers re-process the last prompt tokens) to:

• full-attention layers (35, 39, …, 63) — k_proj raw (post k-norm applied analytically, RoPE applied at serve with true positions) + v_proj raw → the layer's own k-norm/RoPE/cache.
• GatedDeltaNet layers (32, 33, 34, 36, …, 62) — in_proj_qkv raw + in_proj_b + in_proj_a; the layer's own depthwise conv (+silu), β=sigmoid, g=−exp(A_log)·softplus(a+dt_bias), then the true chunked gated-delta recurrence over the prompt → approximate recurrent state S_T (the state the decode steps continue from).

Trained by distillation against the frozen bf16 base (per-token relative-MSE on post-transform quantities, pre-RoPE k targets): 2,000 steps over cached teacher targets (pile-10k, 72×2048 tok), ~1.6 h on one CMP 170HX. Held-out cosines: FA k 0.88 / v 0.81; GDN k 0.90 / v 0.89; GDN S_T relative Frobenius ≈ 0.53 before the stage-2 scan finisher (not included in this checkpoint).

Status / honest limits

• Zero-shot (own-weights init, untrained): 69.8% greedy match vs full baseline @suffix 256; trained checkpoint: 70.8% — at suffix 256 the exact suffix dominates; the untrained lower bound at suffix 1 is 6.8%.
• Text path only; vision tower unused. Single-request prefill protocol.
• Engine reference implementation (HF transformers, CUDA kernels from the transformers/fla stack) ships with the club lane tooling; a vLLM-integrated engine is future work.

Citation / provenance

Mechanism: kishida (nowokay) — LLKVApprox demo, write-up. PixelML club-170hx lane port; tracking: seanphan/pixelml#134.

Measured results (1× CMP 170HX, 180 W, HF bf16)

Held-out target cosines after 2,000-step distillation: FA k 0.88 / v 0.81 · GDN k 0.90 / v 0.89. Training used cached teacher targets (no 27B kernels in the training loop). Engine, bench harnesses, and full receipts ship with the PixelML club-170hx lane tooling (PR: club-170hx#42).