Diffusion-Qwen3.8-27B

🧪 summerMC Diffusion series — autoregressive Qwen models converted into masked diffusion LMs. Experimental research checkpoint: not benchmarked, not for production use.

How it works

Qwen/Qwen3.8-27B is converted into a masked diffusion language model (LLaDA / Dream style). It generates by iterative parallel unmasking, not left-to-right.

• Full-attention layers: causal mask removed (bidirectional).
• Gated DeltaNet layers: forward + reverse scan, out = orig(h) + g ⊙ flip(orig(flip(h))), with a per-channel rev_gate initialised to 0.
• [MASK]: the <|fim_pad|> token with a learned embedding.
• Loss: masked-token cross-entropy with 1/t weighting; shifted prediction (position j from hidden j−1) keeps the AR lm_head aligned.
• Decoding: confidence-threshold parallel unmasking; text after the first EOS is dropped.

Files

Plain AutoModelForCausalLM.from_pretrained gives a causal model, not a diffusion model. Apply the dqwen patch and restore rev_gates / mask_embed as shown below.

Usage

Every model in the series loads the same way through dqwen.hub (shipped in each repo).

pip install -U transformers accelerate huggingface_hub safetensors flash-linear-attention

import sys
from huggingface_hub import snapshot_download
path = snapshot_download("summerMC/Diffusion-Qwen3.8-27B")
sys.path.insert(0, path)
from dqwen.hub import load_pretrained, generate, stream

dlm, tok = load_pretrained(path)
print(generate(dlm, tok, "What is the capital of France?", gen_len=128, thr=0.9))

for ids, newly_committed, step in stream(dlm, tok, "Write a haiku about the sea."):
    pass                                     # undecided tokens are dlm.mask_id

thr is the confidence needed to commit a token at each step: lower is faster, higher is better quality.

⚡ Triton acceleration

load_pretrained enables a Triton fast path by default (dqwen/accel.py, dqwen/triton_kernels.py). Nothing changes in the API: generate, stream and any code calling dlm.hidden(...) get faster.

Safety: before switching on, the fast path is checked against the original layers and the full forward pass (dlm._dqwen_fast_report). If it does not match, or for training / padding masks / BART fusion / batch > 1 / 4-bit / pre-Ampere GPUs, the original PyTorch path is used.

dlm, tok = load_pretrained(path)               # on (default)
dlm, tok = load_pretrained(path, fast=False)   # off
from dqwen import accel
accel.report(dlm)                              # validation numbers
accel.disable(dlm)                             # switch back at runtime

Environment: DQWEN_FAST=0 disables it, DQWEN_CUDA_GRAPH=0 keeps the Triton kernels but disables CUDA graphs. Requires an Ampere-or-newer GPU (A100, L4, RTX 30xx+, H100); flash-linear-attention is strongly recommended.

Training

Requirements

• Hardware: About 56 GB of GPU memory in bf16.
• Packages: transformers (with Qwen3.5 support), accelerate, safetensors, huggingface_hub; optional flash-linear-attention.

Limitations

• Not evaluated: no benchmark has been run; quality is expected to be below the base model.
• No autoregressive generate(): the bidirectional patch runs without a KV/state cache.
• Fragile to library updates: changes to the Qwen implementation in transformers may break the patch.
• Safety: the diffusion adaptation may weaken the base model's safety tuning.
• dqwen/engine.py also has BatchEngine for many prompts in one batched forward pass.
• Slower per token than the AR base on a single prompt (no KV cache, extra reverse scan); throughput gains come from batching.

License

Weights follow the license of the base model(s): Qwen/Qwen3.8-27B. Check their model cards.

References

• LLaDA: Large Language Diffusion Models (Nie et al., 2025)
• Dream 7B: diffusion LM adapted from an AR model (shifted-logits alignment)
• Gated DeltaNet / Qwen3.5 hybrid architecture

Optional block KV / state cache

All three DiffusionQwen models support opt-in block KV and bidirectional DeltaNet state caching. See KV_CACHE.md for usage, approximation trade-offs and validation limits.

from dqwen.kvcache import enable_kv_cache
enable_kv_cache(dlm, refresh=0)  # after load_pretrained; then call generate as usual