Diffusion-Qwen3.5-4B

🧪 summerMC Diffusion series — autoregressive Qwen models converted into masked diffusion LMs. Experimental research checkpoint: not benchmarked, not for production use.

Model	Base	Weights
Diffusion-Qwen3.5-4B (this repo)	Qwen/Qwen3.5-4B	merged weights + BART fusion
Diffusion-Qwen3.5-9B	Qwen/Qwen3.5-9B	merged weights + BART fusion
Diffusion-Qwen3.8-27B	Qwen/Qwen3.8-27B	merged weights

How it works

Qwen/Qwen3.5-4B is converted into a masked diffusion language model (LLaDA / Dream style). It generates by iterative parallel unmasking, not left-to-right.

Full-attention layers: causal mask removed (bidirectional).
Gated DeltaNet layers: forward + reverse scan, out = orig(h) + g ⊙ flip(orig(flip(h))), with a per-channel rev_gate initialised to 0.
[MASK]: the <|fim_pad|> token with a learned embedding.
Loss: masked-token cross-entropy with 1/t weighting; shifted prediction (position j from hidden j−1) keeps the AR lm_head aligned.
Decoding: block-wise low-confidence remasking (diffusion_generate).
BART fusion: facebook/bart-base (FP32, frozen) encodes the draft; a zero-gated cross-attention module injects it into the response embeddings for a refinement pass.

Files

File	Contents
`config.json, model*.safetensors`	merged weights (standard HF format)
`tokenizer files`	Qwen tokenizer
`diffusion_extras.pt`	mask_embed, mask_id, rev_gates
`diffusion_config.json`	base model, mask/EOS tokens and related settings
`dqwen/`	bidirectional patch, loss and sampler
`fusion.pt`	optional BART fusion module

Plain AutoModelForCausalLM.from_pretrained gives a causal model, not a diffusion model. Apply the dqwen patch and restore rev_gates / mask_embed as shown below.

Usage

Every model in the series loads the same way through dqwen.hub (shipped in each repo).

pip install -U transformers accelerate huggingface_hub safetensors flash-linear-attention

import sys
from huggingface_hub import snapshot_download
path = snapshot_download("summerMC/Diffusion-Qwen3.5-4B")
sys.path.insert(0, path)
from dqwen.hub import load_pretrained, generate, stream

dlm, tok = load_pretrained(path)
print(generate(dlm, tok, "What is the capital of France?", gen_len=128, thr=0.9))
# With fusion.pt present, BART refinement runs automatically; return_draft=True gives (draft, final).

for ids, newly_committed, step in stream(dlm, tok, "Write a haiku about the sea."):
    pass                                     # undecided tokens are dlm.mask_id

thr is the confidence needed to commit a token at each step: lower is faster, higher is better quality.

⚡ Triton acceleration

load_pretrained enables a Triton fast path by default (dqwen/accel.py, dqwen/triton_kernels.py). Nothing changes in the API: generate, stream and any code calling dlm.hidden(...) get faster.

	What is fused / removed
Bidirectional Gated DeltaNet	input projections computed once (`in_proj(flip(h)) = flip(in_proj(h))`), forward+reverse conv in one kernel, both scans as one batch-2 `fla` call, `rev_gate` folded into a single `out_proj` GEMM
Norms / MLP / attention	residual add + (1+w) RMSNorm across layers, merged gate/up GEMM + SiLU·up, q/k-norm + partial RoPE, sigmoid output gate
Launch overhead	the whole forward is captured as a CUDA graph per sequence length
Vocab (248k)	`dqwen.accel.select` — fused lm_head + softmax max/argmax/LSE without materialising fp32 logits (for custom samplers)

Safety: before switching on, the fast path is checked against the original layers and the full forward pass (dlm._dqwen_fast_report). If it does not match, or for training / padding masks / BART fusion / batch > 1 / 4-bit / pre-Ampere GPUs, the original PyTorch path is used.

dlm, tok = load_pretrained(path)               # on (default)
dlm, tok = load_pretrained(path, fast=False)   # off
from dqwen import accel
accel.report(dlm)                              # validation numbers
accel.disable(dlm)                             # switch back at runtime

Environment: DQWEN_FAST=0 disables it, DQWEN_CUDA_GRAPH=0 keeps the Triton kernels but disables CUDA graphs. Requires an Ampere-or-newer GPU (A100, L4, RTX 30xx+, H100); flash-linear-attention is strongly recommended.

Training

Item	Value
Base model	Qwen/Qwen3.5-4B
LoRA	merged into the weights
Data	not recorded
Sequence length	not recorded
Steps	demo scale (few steps)

Requirements

Hardware: One CUDA GPU in bf16. flash-linear-attention / causal-conv1d are optional speed-ups (PyTorch fallback otherwise).
Packages: transformers (with Qwen3.5 support), accelerate, safetensors, huggingface_hub; optional flash-linear-attention.

Limitations

Not evaluated: no benchmark has been run; quality is expected to be below the base model.
No autoregressive generate(): the bidirectional patch runs without a KV/state cache.
Fragile to library updates: changes to the Qwen implementation in transformers may break the patch.
Safety: the diffusion adaptation may weaken the base model's safety tuning.
The low-confidence remasking sampler tends to emit EOS too early; ban EOS for the first few tokens or defer EOS commits.
Quality drops as steps decreases (more tokens committed per step).

License

Weights follow the license of the base model(s): Qwen/Qwen3.5-4B. Check their model cards.

References

LLaDA: Large Language Diffusion Models (Nie et al., 2025)
Dream 7B: diffusion LM adapted from an AR model (shifted-logits alignment)
Gated DeltaNet / Qwen3.5 hybrid architecture

Optional block KV / state cache

All three DiffusionQwen models support opt-in block KV and bidirectional DeltaNet state caching. See KV_CACHE.md for usage, approximation trade-offs and validation limits.

from dqwen.kvcache import enable_kv_cache
enable_kv_cache(dlm, refresh=0)  # after load_pretrained; then call generate as usual

summerAIO/Diffusion-Qwen3.5-4B