Kiyo-Diffusion-135M
Masked diffusion language model adapted from DedeProGames/Kiyo-135M: same Llama
weights, bidirectional attention, a new <|mask|> token, the masked-diffusion (MDLM/LLaDA) objective
and the AR shift operation (DiffuLLaMA/Dream recipe). No timestep embedding. Research use only.
python kiyo_diffusion.py generate --model DedeProGames/Kiyo-Diffusion-135M --prompt "The capital of France is"
Diffusion adaptation
The autoregressive Kiyo-135M was converted in place — causal attention replaced by bidirectional
attention, a <|mask|> token added, and the objective switched to masked denoising — then trained
for 250M tokens on the same data mix the base model was pretrained on
(fineweb-edu 45% / dclm-baseline 35% / finemath 10% / stack-v3 10%). Keeping the data distribution
identical is what protects the base knowledge: this is continued pretraining under a new objective,
not a domain shift.
| Setting | Value |
|---|---|
| Tokens | 250M (1908 steps × 131,072 tokens) |
| Learning rate | 1e-4 → 1e-5, cosine, 200 warmup |
| Attention annealing | causal → bidirectional over the first 381 steps (20%) |
| Batch | 131,072 tokens global, sequence length 1024 |
| Precision | bf16 |
| Hardware | 2× RTX 3060 12GB (DDP), ~15k tokens/s |
Benchmark: BananaMind Base Bench 1.1
All 31 training checkpoints were evaluated on BananaMind/BananaMind-Base-Bench-1.1 — 350 four-way text-completion items across 7 categories, reporting a fixed-scale Overall Elo.
| Checkpoint | Overall Elo | Accuracy |
|---|---|---|
| Kiyo-135M, before conversion (official causal metric) | 1125.9 | 67.7% |
| After conversion, 0 training steps | 968.7 | 50.0% |
| Step 500 (65M tokens) | 1054.1 | 60.3% |
| Final, step 1908 (250M tokens) | 1055.4 | 59.1% |
What the curve shows. Opening the attention costs 157 Elo immediately. The first 500 steps (65M tokens) recover 85 of those points — 54% of the gap — and essentially all of the recovery happens there. Across the remaining 1408 steps the Overall Elo stays flat at 1045 ± 10 while the training NELBO keeps falling (4.47 → 4.06): the model keeps getting better at denoising without regaining measurable benchmark capability.
For a 135M-parameter model, diffusion adaptation appears to saturate near 65M tokens, and the
residual ~70 Elo gap looks structural to bidirectional attention at this scale rather than a debt
that more tokens would repay. Per category, world_knowledge and context_tracking recover closest
to the original — bidirectional context helps them — while code_completion suffers most
(0.86 → 0.50), which is unsurprising for a strongly sequential task.
Scoring methodology
A masked diffusion LM has no causal next-token likelihood, so the official metric of this benchmark
(mean conditional token log-probability) cannot be reproduced. Checkpoints are scored instead with a
Monte-Carlo ELBO (n_mc=64), the diffusion analogue used by LLaDA.
To make checkpoints comparable to one another, mask samples are drawn from a seed derived from the item index, so every checkpoint is evaluated against identical masks. Without this, replicate runs of the same checkpoint differed by as much as 16 Elo; with it, replicates are identical.
The dashed baseline in the chart is therefore not an ELBO-equivalent score. It marks where the model started, measured the only way an autoregressive model can be measured. Elo values here are comparable within this chart only, and not with the official leaderboard.