Diffusion-Qwen3.5-9B × BART

🧪 summerMC Diffusion series — autoregressive Qwen models converted into masked diffusion LMs. Experimental research checkpoint: not benchmarked, not for production use.

Model	Base	Weights
Diffusion-Qwen3.5-4B	Qwen/Qwen3.5-4B	merged weights + BART fusion
Diffusion-Qwen3.5-9B (this repo)	Qwen/Qwen3.5-9B	merged weights + BART fusion
Diffusion-Qwen3.8-27B	Qwen/Qwen3.8-27B	merged weights

How it works

Qwen/Qwen3.5-9B is converted into a masked diffusion language model (LLaDA / Dream style). It generates by iterative parallel unmasking, not left-to-right.

Full-attention layers: causal mask removed (bidirectional).
Gated DeltaNet layers: forward + reverse scan, out = orig(h) + g ⊙ flip(orig(flip(h))), with a per-channel rev_gate initialised to 0.
[MASK]: the <|fim_pad|> token with a learned embedding.
Loss: masked-token cross-entropy with 1/t weighting; shifted prediction (position j from hidden j−1) keeps the AR lm_head aligned.
Decoding: draft by block-wise remasking, then BART-guided refinement of the lowest-confidence 30%.
BART fusion: facebook/bart-base (FP32, frozen) encodes the draft; a zero-gated cross-attention module injects it into the response embeddings for a refinement pass.

Files

File	Contents
`config.json, model-0000X-of-00004.safetensors`	merged bf16 weights (LoRA merged; [MASK] embedding written into the mask-token row)
`tokenizer.json, tokenizer_config.json, chat_template.jinja`	Qwen tokenizer
`dqwen_rev_gates.safetensors`	DeltaNet reverse gates (metadata: mask_id, mask_token, step)
`fusion.pt`	BartFusion state_dict
`dqwen/`	bidirectional patch, sampler, BART fusion and training code

Plain AutoModelForCausalLM.from_pretrained gives a causal model, not a diffusion model. Apply the dqwen patch and restore rev_gates / mask_embed as shown below.

Usage

Every model in the series loads the same way through dqwen.hub (shipped in each repo).

pip install -U transformers accelerate huggingface_hub safetensors flash-linear-attention

import sys
from huggingface_hub import snapshot_download
path = snapshot_download("summerMC/Diffusion-Qwen3.5-9B")
sys.path.insert(0, path)
from dqwen.hub import load_pretrained, generate, stream

dlm, tok = load_pretrained(path)
print(generate(dlm, tok, "What is the capital of France?", gen_len=128, thr=0.9))
# With fusion.pt present, BART refinement runs automatically; return_draft=True gives (draft, final).

for ids, newly_committed, step in stream(dlm, tok, "Write a haiku about the sea."):
    pass                                     # undecided tokens are dlm.mask_id

thr is the confidence needed to commit a token at each step: lower is faster, higher is better quality.

⚡ Triton acceleration

load_pretrained enables a Triton fast path by default (dqwen/accel.py, dqwen/triton_kernels.py). Nothing changes in the API: generate, stream and any code calling dlm.hidden(...) get faster.

	What is fused / removed
Bidirectional Gated DeltaNet	input projections computed once (`in_proj(flip(h)) = flip(in_proj(h))`), forward+reverse conv in one kernel, both scans as one batch-2 `fla` call, `rev_gate` folded into a single `out_proj` GEMM
Norms / MLP / attention	residual add + (1+w) RMSNorm across layers, merged gate/up GEMM + SiLU·up, q/k-norm + partial RoPE, sigmoid output gate
Launch overhead	the whole forward is captured as a CUDA graph per sequence length
Vocab (248k)	`dqwen.accel.select` — fused lm_head + softmax max/argmax/LSE without materialising fp32 logits (for custom samplers)

Safety: before switching on, the fast path is checked against the original layers and the full forward pass (dlm._dqwen_fast_report). If it does not match, or for training / padding masks / BART fusion / batch > 1 / 4-bit / pre-Ampere GPUs, the original PyTorch path is used.

dlm, tok = load_pretrained(path)               # on (default)
dlm, tok = load_pretrained(path, fast=False)   # off
from dqwen import accel
accel.report(dlm)                              # validation numbers
accel.disable(dlm)                             # switch back at runtime

Environment: DQWEN_FAST=0 disables it, DQWEN_CUDA_GRAPH=0 keeps the Triton kernels but disables CUDA graphs. Requires an Ampere-or-newer GPU (A100, L4, RTX 30xx+, H100); flash-linear-attention is strongly recommended.

Training

Item	Value
Base model	Qwen/Qwen3.5-9B
LoRA	r=8, alpha=16 (4-bit NF4 QLoRA, bf16 compute), merged into bf16 weights
Data	256 samples from HuggingFaceTB/smoltalk (`all`)
Sequence length	256 (prompt ≤ 128)
Steps	Stage 1: 10 (LoRA, gates, mask embedding); Stage 2: 10 (fusion only)

Requirements

Hardware: About 18 GB of GPU memory in bf16 (A100 / L4 class); 4-bit loading fits smaller GPUs.
Packages: transformers (with Qwen3.5 support), accelerate, safetensors, huggingface_hub; optional flash-linear-attention.

Limitations

Not evaluated: no benchmark has been run; quality is expected to be below the base model.
No autoregressive generate(): the bidirectional patch runs without a KV/state cache.
Fragile to library updates: changes to the Qwen implementation in transformers may break the patch.
Safety: the diffusion adaptation may weaken the base model's safety tuning.
The reverse gates live in dqwen_rev_gates.safetensors, not in the HF weights; load_rev_gates must be called after patch_bidirectional.

License

Weights follow the license of the base model(s): Qwen/Qwen3.5-9B, facebook/bart-base. Check their model cards.

References

LLaDA: Large Language Diffusion Models (Nie et al., 2025)
Dream 7B: diffusion LM adapted from an AR model (shifted-logits alignment)
Gated DeltaNet / Qwen3.5 hybrid architecture

Optional block KV / state cache

All three DiffusionQwen models support opt-in block KV and bidirectional DeltaNet state caching. See KV_CACHE.md for usage, approximation trade-offs and validation limits.

from dqwen.kvcache import enable_kv_cache
enable_kv_cache(dlm, refresh=0)  # after load_pretrained; then call generate as usual

summerAIO/Diffusion-Qwen3.5-9B