Qwen3.6 35B-A3B Distill — DFlash2 Draft
A DFlash2 block-diffusion draft model for
freakyskittle/qwen36-35b-a3b-distill.
It speculatively drafts blocks of tokens that the target verifies in parallel,
giving multi-token acceptance per decoding step.
This is not a text-generation model — it produces draft blocks only and must be paired with the target model and a DFlash2-aware serving stack.
Architecture
| Type | DFlash2DraftModel (bidirectional block-diffusion draft) |
| Parameters | ~432M |
| Layers | 5 (sliding attention, window 2048) |
| Hidden | 2048 |
| Block size | 8 |
| Mask token id | 248077 |
| Target taps | hidden layers [1, 10, 19, 28, 37] of the 40-layer target |
| Candidate selector | rank 256, top-k 16 |
| Conv | dynamic grouped causal, kernel 2, group 16 |
The draft borrows the target's input embedding and LM head at serve time —
it ships only its own weights (model.safetensors, config.json).
Training
- Target hidden states captured from the merged model in bf16 over the same distillation corpus used for the LoRA (private GLM distillation + code / security / benchmark mix, rendered through the chat template).
- Masked-block cross-entropy over all 8 block positions plus a teacher-forced candidate-selector loss (λ = 0.5).
- Trained in fp32 (bf16 was numerically unstable), saved as bf16.
- 10,000 steps @ 0.57s/step on an L4; final window loss 3.92, selector loss 14.4.
Held-out block containment
top1 22.6% top3 34.0% top8 44.9% top16 53.5%
Containment = fraction of masked block positions whose true token appears in the draft's top-k — the metric that maps to speculative acceptance.
GGUF
Qwen36-35B-A3B-Distill-DFlash2-F16.gguf (864 MB) ships in the sibling repo
freakyskittle/qwen36-35b-a3b-distill-GGUF
alongside the main-model quants — converted with llama.cpp's DFlash2DraftModel
support (--target-model-dir against the merged repo).
Usage
Requires a serving stack with DFlash2 support: the
dflash Python package plus a compatible
backend — llama.cpp (PR #27342),
SGLang, or vLLM with the DFlash patch sets referenced in the dflash README.
The reference draft repos (e.g. z-lab/Qwen3.8-27B-DFlash2) show the expected
serving configuration.
Reproduction
Capture + training scripts live in the Unsloth fork under
examples/distillation_sft/dflash/:
Jackson57279/unsloth.
Limitations
- Draft model only — cannot generate text by itself.
- Tuned to this specific merged checkpoint; do not pair with other weights.
- Trained on a bounded hidden-state corpus; containment will improve with a larger capture.