trillionlabs/Zoro

English | 한국어

Zoro is a two-stream block-diffusion language model (research checkpoint). It is produced by continued SFT with the full-mask recipe — 100% of each block masked (single view, no complementary mask) at block size bd=8, fresh optimizer, LR 5e-5, ~2k steps — starting from the base SFT checkpoint (v6, step_18000).

Code

Training and serving code lives in trillion-labs/trida-stack — the two-stream block-diffusion training stack (train/) and the HybridDiffusion SGLang serving fork (inference/). See the getting-started notebooks there for first-run training and inference.

How to serve

Best served self-speculative (draft with the diffusion stream, verify with the clean/AR stream): verify-guaranteed AR quality, and — because full-mask training shapes exactly the single-shot full-block prediction a self-spec draft needs — a higher acceptance length than the base, so it's the fastest self-spec at low concurrency. It also serves in plain AR/causal mode. Pure iterative diffusion decoding is not supported by this recipe (the shift decoder's partial-mask states are untrained). Runs on the HybridDiffusion SGLang fork. mask_id = 248077, vocab 248078.

Accuracy

Concurrency 1, threshold 0.9, reasoning on. A vanilla Qwen3.5-4B baseline is being added.

Serving speed & parallel commit (concurrency 1)

Block-diffusion self-spec commits multiple tokens per forward pass at verify-guaranteed AR quality. Zoro's full-mask training makes it a better drafter, so it accepts more per step and runs faster than the base.

Per-benchmark commit rate (Zoro self-spec, tokens/forward): GSM8K 3.10, BFCL 2.33, MMLU-Pro 2.33.

License

Model weights are Apache-2.0. The base model (Qwen/Qwen3.5-4B) is Apache-2.0, and no use restriction propagates from it.