AEON DFlash 2 for Qwen3.8-27B AEON Ultimate

v1.0 (lk4) · released 2026-10-05 · Early Release · Gated Access

Measured with the previous stage (lk3): +21% faster than the stock DFlash 2 drafter at T=0.6, +35% at T=1.0, 3.24× faster than plain decoding for one user.† Lossless by construction.

Measured on one NVIDIA DGX Spark with the published AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED build, against stock z-lab/Qwen3.8-27B-DFlash2 (BF16).

† The "vs plain decoding" multiples are unpaired: AEON ran 537 prompts at T=0.6, the no-speculation baseline 72 prompts (4 per category, about 10.1 tok/s in every category). "Lossless" holds by construction: rejection sampling has the target verify every drafted token. Our small accuracy check is not proof of it; that check was dominated by 512-token output caps on the no-speculation baseline (see Notes and caveats).

Read this first. The served speed numbers on this page were measured with lk3, the previous training stage. This release (lk4) accepts +0.7% more drafted tokens than lk3 offline ([+0.47, +1.00], 95% CI), so expect equal or slightly better speed. lk4's own served numbers are coming: a full category × concurrency benchmark is next on the DGX Spark. More extensive tests and data to come.

Weights update, 2026-10-05. Within the first hours of release the v1.0 weights were updated from training step 690 to step 753 (lk4, chosen by the release rule's tie-break). Offline acceptance is identical (+0.70% vs lk3, +4.87% vs stock z-lab). If you downloaded earlier, re-download; the new sha256 values are under What's in this repo.

Built on DFlash 2 by Inco AI & z-lab.

AEON DFlash 2 drafts up to nine tokens in one parallel pass. The 27B model checks them all at once, keeps what it agrees with, and moves on. It still decides every token. You just get them sooner.

It is a speculative-decoding drafter, not a standalone chat model. It runs next to the target it was trained for, AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED, and it was fine-tuned from z-lab/Qwen3.8-27B-DFlash2 (a mirror of incoai/Qwen3.8-27B-DFlash2, Apache-2.0) on how that target is actually served.

Two precisions, two folders. The repo root holds the drafter as an NVFP4 W4A16 pack (4-bit MLP weights, 16-bit activations, 1.93 GB) for NVIDIA Blackwell with vLLM. The BF16 weights (3.85 GB) live only in the bf16/ folder, for every other architecture and engine. See Which weights do I use? and Quickstart by architecture.

Access

Request access on the model page. Requests are reviewed every few hours. Supporters: include the community access word from the pinned post for automatic approval.

Initial gated access for Patreon supporters: early-access post · become a member.

At a glance

Why it's great

• Trained on the model you actually serve. It learned the published NVFP4-MIXED target's real served output distribution at its serving sampling settings (T=0.6, top-p 0.95, top-k 20), thinking and tool calls included, with an objective that directly maximizes how many drafted tokens the target keeps.
• Native NVFP4 (repo root). Trained with NVFP4 quantization-aware training on the exact grid it serves on. The published W4A16 pack in the repo root matches the training grid bit for bit (export check: every FP4 code and scale equal). bf16/ holds the same values expanded to BF16.
• Built for 9 drafts per step (block 10), where z-lab's published default is 7.
• Tuned for DGX Spark. A per-concurrency draft-length lattice plus probabilistic drafting.
• Drop-in. The bf16/ copy has the same 81 tensors (names, shapes, dtypes) as z-lab/Qwen3.8-27B-DFlash2, so it loads wherever that drafter runs.

Recommended configuration (the measured production config)

Which weights do I use?

This repo holds one drafter in two precisions. Download only the one your hardware needs.

• The BF16 weights live only in bf16/. The root model.safetensors is the NVFP4 W4A16 pack, not BF16. Both folders hold the same trained drafter: bf16/ is the root pack expanded to BF16, with 0 differing elements.
• What W4A16 means here. The 15 MLP linears' weights are stored in 4 bits and expanded on the fly by vLLM's Marlin kernel (FlashInfer CuTe-DSL on B200 / B300 in vLLM ≥ 0.30). The math runs with BF16 activations, not on FP4 tensor cores. The pack halves the drafter's bytes, which helps on bandwidth-bound Blackwell boxes such as the DGX Spark. At load vLLM may log "Your GPU does not have native support for FP4 computation…"; that line is generic and expected, on Blackwell too.
• Older NVIDIA GPUs. On Ampere, Ada and Hopper (A100, RTX 30 / 40, L40S, H100 / H200), vLLM can also load the root pack through Marlin W4A16 (checked against vLLM's source; untested). Use bf16/ there: it is the path documented below. Turing (T4) is not supported, because the drafter needs BF16.
• AMD ROCm: vLLM cannot load the root pack (its NVFP4 Marlin kernel is CUDA-only). Use bf16/.

Which files are where:

AEON-7/AEON-DFlash2-Qwen3.8-27B
├── model.safetensors          NVFP4 W4A16 drafter, 1.93 GB    ← NVIDIA Blackwell + vLLM
├── config.json                with quantization_config (modelopt_mixed, W4A16_NVFP4)
├── hf_quant_config.json       NVFP4 pack description
├── FIDELITY_GATE.json         pack checks
├── LOAD_SHAPE_AUDIT.json
├── bf16/
│   ├── model.safetensors      BF16 drafter, 3.85 GB           ← everything else
│   ├── config.json            same config, without quantization_config
│   ├── DEQUANT_REPORT.json    NVFP4 → BF16 exactness report
│   └── README.md, LICENSE, NOTICE
├── configs/  scripts/  recipes/  docs/  images/
└── README.md, AGENTS.md, CHANGELOG.md, CREDITS.md, EVOLUTION.md, LICENSE, NOTICE

Download only the folder you need. The files are gated: request access first, then hf auth login.

# NVIDIA Blackwell: repo root, NVFP4 W4A16 (~1.9 GB). Drafter dir = ~/models/aeon-dflash2
hf download AEON-7/AEON-DFlash2-Qwen3.8-27B --local-dir ~/models/aeon-dflash2 --exclude "bf16/*"

# Everything else: bf16/ only (~3.85 GB). Drafter dir = ~/models/aeon-dflash2/bf16
hf download AEON-7/AEON-DFlash2-Qwen3.8-27B --local-dir ~/models/aeon-dflash2 --include "bf16/*"

With --include "bf16/*" the weights land in ~/models/aeon-dflash2/bf16/. Point your engine at that subfolder, not at ~/models/aeon-dflash2, which then holds no weights.

Quickstart by architecture

On every vLLM path: VLLM_USE_V2_MODEL_RUNNER=1, K=9 and never above, the per-concurrency lattice, TRITON_ATTN inside --speculative-config, probabilistic draft sampling, and no MTP at the same time.

A. NVIDIA Blackwell: NVFP4 W4A16 (root)

DGX Spark (GB10): tested.

hf download AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED --local-dir ~/models/aeon-27b
hf download AEON-7/AEON-DFlash2-Qwen3.8-27B --local-dir ~/models/aeon-dflash2 --exclude "bf16/*"
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-18-v0.29.0-omni

Serve with the full command in Quickstart: one DGX Spark, step 2. It mounts the repo root (~/models/aeon-dflash2, the NVFP4 W4A16 pack) as /draft. Weights total about 26.6 GB (24.7 GB target + 1.93 GB drafter).

curl -sf http://localhost:8000/health && echo OK
docker logs aeon-dflash2 2>&1 | grep -E "DFlash2|MarlinNvFp4LinearKernel" | head   # drafter loaded on the W4A16 (Marlin) path
curl -s http://localhost:8000/metrics | grep -E '^vllm:spec_decode_num_(drafts|accepted_tokens)_total'
# τ = 1 + accepted ÷ drafts. Measured (lk3): about 4.1 at 1 user, T=0.6

RTX 5090 / RTX PRO 6000 (sm_120): untested, being validated (recipes/rtx-5090.md). The Spark image is arm64 and won't run there. The NVFP4-MIXED card's RTX image is ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:latest; it has not yet been checked for DFlash 2 support, which needs vLLM 0.28.0 or newer. Use the same root download, /draft mount, environment and --speculative-config as the Spark command, with the memory settings from the target card's RTX seats. On a 32 GB RTX 5090 the weights alone take about 26.6 GB, so little is left for the KV cache; the target card's 5090 seat uses MTP instead.

B200 / B300 (sm_100 / sm_103): untested. No AEON image is published for them. Stock vLLM ≥ 0.30 runs the root pack's MLPs on FlashInfer CuTe-DSL W4A16 when FlashInfer provides that kernel (otherwise Marlin). Serving the NVFP4-MIXED target on stock vLLM may also need the target's ModelOpt patch (modelopt-54367.py, see its card).

B. NVIDIA Hopper / Ampere / Ada: bf16/ (untested)

For H100, H200, A100, L40S, RTX 6000 Ada, RTX 30 / 40 and other sm_80+ GPUs. The NVFP4-MIXED target is a Blackwell build, so pair bf16/ with the BF16 target. Use stock vLLM 0.28.0 or newer: DFlash 2 first shipped in 0.28.0, and the latest release is 0.31.0.

pip install -U huggingface_hub "vllm>=0.28.0"
hf auth login
hf download AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 --local-dir ~/models/aeon-27b-bf16
hf download AEON-7/AEON-DFlash2-Qwen3.8-27B --local-dir ~/models/aeon-dflash2 --include "bf16/*"

DRAFT="$HOME/models/aeon-dflash2/bf16"     # the BF16 drafter: the bf16/ subfolder, not the root
VLLM_USE_V2_MODEL_RUNNER=1 vllm serve ~/models/aeon-27b-bf16 \
  --served-model-name aeon --port 8000 \
  --dtype bfloat16 --max-model-len 65536 --gpu-memory-utilization 0.90 \
  --no-enable-prefix-caching \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --trust-remote-code \
  --speculative-config '{"method":"dflash","model":"'"$DRAFT"'","num_speculative_tokens":9,"num_speculative_tokens_per_batch_size":[[1,2,9],[3,8,7],[9,12,6],[13,16,4]],"attention_backend":"TRITON_ATTN","draft_sample_method":"probabilistic"}' \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'
# Two GPUs (for example 2x 48 GB): add --tensor-parallel-size 2

• Untested. This exact pairing (stock vLLM, BF16 target, bf16/ drafter) has not been run yet; stock-vLLM validation is tracked in recipes/vllm-upstream.md. Acceptance with the BF16 target is not measured: the drafter was trained on the NVFP4-MIXED target's outputs.
• Prefix caching off: every vLLM release so far (through 0.31) needs --no-enable-prefix-caching with this hybrid Gated-DeltaNet target (vllm#58894; fix pending in vllm#55601).
• Don't add MTP. The BF16 target card's own command uses "method":"mtp"; use one speculative method at a time.
• Memory (weights only, estimates): 55.6 GB target + 3.85 GB drafter ≈ 59.4 GB. H200 (141 GB): comfortable. H100 / A100 80 GB: fits on one GPU with roughly 10 GB left for the KV cache at 0.90, so keep the context modest (65,536 above is a conservative suggestion, not a measurement). 48 GB cards: tensor parallel 2, about 29.7 GB of weights per GPU. The target's KV cache takes about 65.5 KB per token in BF16, plus a fixed Gated-DeltaNet state per running request.
• Add --api-key <secret> if other people can reach the port. With data parallel above 1, vLLM ignores the lattice and uses a fixed K=9.

curl -sf http://localhost:8000/health && echo OK
curl -s http://localhost:8000/metrics | grep -E '^vllm:spec_decode_num_(drafts|accepted_tokens)_total'
# τ = 1 + accepted ÷ drafts. Well below 3 on normal prompts: check that "model" points at .../bf16
# and that VLLM_USE_V2_MODEL_RUNNER=1 is set (the startup log should name DFlash2DraftModel)

C. AMD ROCm: bf16/ (untested on AMD)

Use the kit that ships with the ROCm target, AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-MXFP4-MXFP6-ROCm (gated). Its card has the full setup, the pinned vLLM 0.31.0 ROCm image and a recipe per GPU. Its Recipe A (2x Radeon AI PRO R9700, TP=2) already runs DFlash 2 with the right settings: K=9 plus the lattice, TRITON_ATTN inside and outside the speculative config, the V2 runner and prefix caching off. The kit bundles an earlier BF16 drafter (lk3) in its dflash2/ folder. To run this release (lk4), download bf16/ from this repo and point the kit at it with AEON_DFLASH:

hf auth login
hf download AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-MXFP4-MXFP6-ROCm --local-dir ~/models/aeon-mxfp4-rocm
hf download AEON-7/AEON-DFlash2-Qwen3.8-27B --local-dir ~/models/aeon-dflash2 --include "bf16/*"
export AEON_DIR="$HOME/models/aeon-mxfp4-rocm"
# Set AEON_CACHE, AEON_IMAGE and RENDER_GID as in the ROCm card's "Common setup" step, then:
AEON_DFLASH="$HOME/models/aeon-dflash2/bf16" bash "$AEON_DIR/scripts/serve_2x_r9700_dflash2.sh"

• Other AMD GPUs: python3 "$AEON_DIR/scripts/detect_setup.py" picks the recipe. Recipes K (4x R9700) and Q (2x W7900) also use DFlash 2: in their plain docker run commands, replace the -v "$AEON_DIR/dflash2":/draft:ro mount with -v "$HOME/models/aeon-dflash2/bf16":/draft:ro.
• Never the repo root on ROCm: vLLM cannot load the NVFP4 pack there.
• Memory (weights only): 23.7 GB target + 3.85 GB drafter ≈ 27.5 GB.
• Untested: lk4 has not been run on AMD hardware. The kit's measured τ of 3.73 at T=0.6 was with its bundled lk3 drafter.

curl -sf http://localhost:8000/health && echo OK
# Startup log: "Resolved architecture: DFlash2DraftModel". Under load: "SpecDecoding metrics: Mean acceptance length"
# of about 3 or more. About 1 with several requests in flight means the draft is not on TRITON_ATTN.

D. Other engines: bf16/

hf download AEON-7/AEON-DFlash2-Qwen3.8-27B --local-dir ~/models/aeon-dflash2 --include "bf16/*"
# Drafter dir for every engine below: ~/models/aeon-dflash2/bf16

• SGLang, llama.cpp, TensorRT-LLM, TensorFold and the MLX tools all take bf16/, with a draft width of 9 (block 10). Versions, flags and commands for each: Use it with any inference engine. None has been tested with an AEON target yet.
• Target: the BF16 target for SGLang and TensorRT-LLM, and a GGUF conversion of it for llama.cpp. TensorFold serves only its own Qwen3.8-27B target formats, not the AEON checkpoints as shipped. On Apple Silicon no AEON MLX build is published, so you would convert the BF16 target with mlx-lm yourself (untested).
• Check: SGLang, llama-server and trtllm-serve answer curl -sf http://localhost:<port>/health. Then read the acceptance length from the engine's speculative-decoding log or metrics: about 3 or more on normal prompts at T=0.6. Much lower usually means the drafter path or the draft width is wrong.

Speed: measured on one DGX Spark (lk3, the previous stage)

AEON DFlash 2 (lk3, tuned config: K=9 + lattice + probabilistic drafting) vs stock z-lab DFlash 2 (BF16, unmodified, K=7, its published config) vs no speculation. Same DGX Spark, same published target build, same image and serving flags; only the drafter and --speculative-config change.

One user, all categories pooled

τ = acceptance length: tokens produced per forward pass of the 27B target. Decode tok/s per request, thinking included.

† "vs no speculation" multiples are unpaired. AEON ran 537 prompts at T=0.6 (180 at T=1.0 and T=0); the no-speculation baseline ran only 72 prompts at T=0.6 (4 per category) and 16 at T=1.0 and T=0 (2 per category). Its decode speed is nearly constant (about 10.1 tok/s in every category), so each multiple divides by that category's no-speculation mean rather than pairing prompt by prompt.

By category, T=0.6 (sorted by gain; every category is faster, every interval above zero)

By category, T=1.0 (z-lab task set plus tools, writing and multilingual)

By category, T=0 (greedy)

At T=0 the gain is smallest on creative writing, where the interval crosses zero.

Concurrency (T=0.6; 48 math, coding and writing prompts, outputs capped at 512 tokens; two replicate runs per arm)

The 1-user row is the pooled T=0.6 result above (all 537 prompts). The 4- and 16-user rows are closed-loop runs over the 48-prompt set; "vs stock z-lab" there is a ratio of the two arms' totals.

The chart plots steady-state tok/s (ramp-up and drain excluded): 102 / 247 at 4 / 16 users for AEON, 84 / 208 for stock z-lab. The table gives whole-run totals.

Where the gain comes from (T=0.6, the 196 prompts all three arms ran)

The serving config does most of the work, and you can apply it to the stock drafter too (docs/HOSTING.md). Training adds a further, statistically clear +3.5% on top. lk4 adds about +0.7% more acceptance over lk3 (offline, below).

Full method, datasets and caveats: docs/BENCHMARKS.md.

Acceptance, this release (lk4 = S5 step 753)

Offline acceptance on the published NVFP4-MIXED build: 213 held-out windows of the target's own served traffic, weighted by workload type, with item-cluster bootstrap 95% confidence intervals. Change in acceptance length τ. lk4 is the step-753 save of training stage S5, chosen by the release rule's tie-break (step 690, which shipped for the first hours of v1.0, scored the same overall).

For reference, lk3 measured +4.13% [+3.69, +4.66] over stock z-lab BF16 on the same windows.

What to expect served: lk3's served speed above, plus roughly lk4's +0.7% acceptance gain. That is an expectation, not a measurement. lk4's own served benchmark is next.

Notes and caveats

• Stock z-lab needs TRITON_ATTN on this image. On the AEON vLLM 0.29 image, the stock z-lab drafter would not boot until its --speculative-config included "attention_backend":"TRITON_ATTN". Every z-lab arm above ran that way, with everything else at its published config (K=7, greedy draft sampling).
• Accuracy parity: no significant difference, but the check is small and is not proof. Paired against no speculation on 34 scored prompts at T=0.6 (drawn from GSM8K, MATH-500, BBH, BFCL, MMLU-Pro, MGSM, HotpotQA and NQ-Open): AEON −2.9 points [−11.8, +5.9], stock z-lab +2.9 [−8.8, +14.7], McNemar p = 1.00 for both. That is too small to rule out small effects, and it was dominated by 512-token output caps on the no-speculation baseline: with thinking on, many of its answers were cut off, which is why its accuracy is only 35.3%. It says little about quality either way. "Lossless" rests on construction (rejection sampling has the target verify every token), not on this check. A larger accuracy check with full-length outputs is part of the next benchmark.
• Not byte-identical, even at T=0. No arm reproduced the no-speculation text exactly at T=0 (0 of 16 for both AEON and stock z-lab). A different number of tokens per step changes floating-point rounding and can flip near-ties. The distribution is unchanged.
• BF16 SSM state cache: not recommended. --mamba-ssm-cache-dtype bfloat16 gave +15.5% total throughput at 16 users (248 vs 215 tok/s, one replicate). Quality: inconclusive overall (−1.8 pts pooled over 388 scored items, 95% CI [−4.4, +0.5], crosses zero); one chained-IFEval subset dropped 12.5 pts (95.8% → 83.3%, n=48, McNemar p=0.07); not recommended. Keep the FP32 default.
• Home-field advantage, stated plainly. z-lab's drafter was trained for stock Qwen3.8-27B in BF16. Here it drafts for a modified NVFP4-MIXED model it never saw. Ours was trained for exactly this target.
• Different from z-lab's published numbers. Their card uses SGLang on an H200 with the stock target. Different hardware, engine and target.

More tests and data coming

• Full lk4 benchmark on the DGX Spark: every category at 1, 4, 8 and 16 concurrent users, with acceptance, tok/s, time to first token, time per output token, and prefill at 1K, 8K and 32K-token inputs.
• Continued training: the next stage is underway, on fresh on-policy data from the published build. A new version is published only when it shows an overall acceptance gain, and every version is tracked in EVOLUTION.md.
• More platforms: RTX 5090 / RTX PRO 6000 and stock upstream vLLM are being validated (recipes/).

Quickstart: one DGX Spark

This is quickstart A in full: NVIDIA Blackwell, with the NVFP4 W4A16 drafter from the repo root. Other hardware: Quickstart by architecture.

Using an AI agent to deploy? Point it at AGENTS.md.

1. Get access and download. Request access on the model page first, then:

python3 -m venv ~/hf && . ~/hf/bin/activate && pip install -U huggingface_hub
hf auth login
hf download AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED --local-dir ~/models/aeon-27b
hf download AEON-7/AEON-DFlash2-Qwen3.8-27B --local-dir ~/models/aeon-dflash2 --exclude "bf16/*"
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-18-v0.29.0-omni

Use --local-dir: Hugging Face cache snapshot folders are symlinks and break inside Docker. The image is public (arm64). --exclude "bf16/*" downloads the repo root (the NVFP4 W4A16 drafter, 1.93 GB) and skips the 3.85 GB BF16 copy in bf16/, which Blackwell doesn't need.

2. Serve. /draft = this repo's root (the NVFP4 W4A16 pack, not bf16/). This matches the measured production config, except: default memory 0.80 (we measured at 0.57 with other services on the box), aliases omitted (extra --served-model-name entries), restart policy optional:

GPU_UTIL=0.80   # dedicated Spark. Use 0.57 if other GPU services share the box (our measured setting).

docker run -d --name aeon-dflash2 --gpus all --network host \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e VLLM_ALLREDUCE_USE_FLASHINFER=0 \
  -v ~/models/aeon-27b:/model:ro \
  -v ~/models/aeon-dflash2:/draft:ro \
  -v ~/models/aeon-27b/modelopt-54367.py:/usr/local/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/modelopt.py:ro \
  --entrypoint vllm ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-18-v0.29.0-omni \
  serve /model \
  --served-model-name aeon --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization $GPU_UTIL \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 16384 \
  --kv-cache-dtype fp8 \
  --enable-chunked-prefill \
  --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image":4,"video":2}' \
  --attention-backend TRITON_ATTN \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --trust-remote-code \
  --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":9,"num_speculative_tokens_per_batch_size":[[1,2,9],[3,8,7],[9,12,6],[13,16,4]],"attention_backend":"TRITON_ATTN","draft_sample_method":"probabilistic"}' \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• modelopt-54367.py ships in the target's model folder. The measured container mounts it; the target's card notes the same patch is already baked into this image tag, so the mount is a harmless pin.
• Memory: 0.80 on a dedicated Spark (the target card's setting). We measured at 0.57 because speech services share our Spark. Don't go above 0.80: running out of unified memory can freeze the machine.
• Never K > 9. Don't combine with MTP. No --quantization flag.
• Add --restart unless-stopped to start again after a reboot, and --api-key <secret> if other people can reach port 8000.

3. Health check. The first boot takes several minutes (docker logs -f aeon-dflash2).

curl -sf http://localhost:8000/health && echo OK
curl -s http://localhost:8000/v1/models          # lists "aeon" when ready

4. Test request.

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"aeon","messages":[{"role":"user","content":"Explain speculative decoding in two sentences."}],"max_tokens":512,"chat_template_kwargs":{"enable_thinking":false}}'

5. Measure τ. On an otherwise idle server:

scripts/test_request.sh        # streams a thinking-on request; prints TTFT, decode tok/s and τ
curl -s http://localhost:8000/metrics | grep -E '^vllm:spec_decode_num_(drafts|accepted_tokens)_total'
# τ = 1 + accepted ÷ drafts (take the difference before and after a request)

At 1 user and T=0.6 we measured τ ≈ 4.1 pooled over all categories (lk3), from about 3 on roleplay and creative writing to over 6 on tool calls. Much lower than 3 on normal prompts means something is off: check the /draft mount and "draft_sample_method":"probabilistic".

Docker Compose, Python client and troubleshooting: docs/QUICKSTART.md. Recipes per platform: recipes/.

Use it with any inference engine

The drop-in rule: any engine that runs z-lab/Qwen3.8-27B-DFlash2 runs this drafter. Point its drafter path at this repo's bf16/ folder instead and set the draft width to 9. Each engine names that setting differently (see the table).

Which folder to use:

• Repo root (NVFP4 W4A16): vLLM on NVIDIA Blackwell. Verified on DGX Spark with the AEON vLLM 0.29 image; stock upstream vLLM is being validated.
• bf16/ (exact BF16 copy of the same weights, the only BF16 weights in the repo): everything else, including vLLM on NVIDIA Hopper / Ampere / Ada and on AMD ROCm, SGLang (the NVFP4 pack is untested there), llama.cpp (convert from bf16/), TensorRT-LLM, TensorFold (which re-packs to 4-bit at load) and MLX.

vLLM (NVIDIA Blackwell: the repo root; NVIDIA Hopper / Ampere / Ada and AMD ROCm: bf16/)

VLLM_USE_V2_MODEL_RUNNER=1 vllm serve <AEON target dir> \
  --speculative-config '{"method":"dflash","model":"<drafter dir>","num_speculative_tokens":9,"num_speculative_tokens_per_batch_size":[[1,2,9],[3,8,7],[9,12,6],[13,16,4]],"attention_backend":"TRITON_ATTN","draft_sample_method":"probabilistic"}'
# Stock upstream vLLM (every release so far, through 0.31) with this hybrid-GDN target: add --no-enable-prefix-caching
# Hopper / Ampere / Ada: point "model" at the bf16/ folder and use the BF16 target (Quickstart B)
# ROCm: also pass --attention-backend TRITON_ATTN and point "model" at the bf16/ folder (Quickstart C)

SGLang

The block (verify window) is 10. If you leave the flag unset, SGLang reads it from the config. z-lab's snippet hard-codes 8, which would run this drafter narrower than it was trained.

python -m sglang.launch_server --model-path <AEON target dir> \
  --speculative-algorithm DFLASH --speculative-draft-model-path <drafter>/bf16 \
  --speculative-num-draft-tokens 10

llama.cpp

Convert the drafter from bf16/. Don't reuse z-lab's DFlash2 GGUFs: they hold z-lab's weights with block 8.

python convert_hf_to_gguf.py <drafter>/bf16 --target-model-dir <AEON target HF dir> \
  --outtype bf16 --outfile aeon-dflash2.gguf
llama-server -m <AEON target>.gguf -md aeon-dflash2.gguf \
  --spec-type draft-dflash --spec-draft-n-max 9 -fa on --jinja

TensorRT-LLM (trtllm-serve <target dir> --config spec.yaml)

speculative_config:
  decoding_type: DFlash
  max_draft_len: 9
  speculative_model: <drafter>/bf16

TensorFold

TensorFold's DFlash2 loader reads this drafter and re-packs its linears to 4-bit at load. This can lower acceptance, but not output fidelity.

tensorfold serve <supported target> --drafter <drafter>/bf16

Only vLLM has been run with an AEON target. The other engines run z-lab's Qwen3.8-27B DFlash2, so by format they run this drafter too, but we haven't tested them with AEON targets yet.

Where it runs

What's in this repo

What it is

• A DFlash 2 draft model. It reads hidden states from five target layers and proposes up to 9 tokens per step; the target checks them in one forward pass.
• Fine-tuned from z-lab/Qwen3.8-27B-DFlash2 by Inco AI / z-lab (Apache-2.0). Its config differs only in block_size 10, causal: false and draft_vocab_size.
• NVFP4 weights (W4A16) in the repo root; BF16 in bf16/. In the root pack the 15 MLP linears hold FP4 weights. Attention and every other layer stay BF16. Activations are BF16, and vLLM's Marlin kernel runs the math (FlashInfer CuTe-DSL on B200 / B300 in vLLM ≥ 0.30): weight-only FP4, not FP4 tensor-core math.
• Lossless by construction. Speculative decoding with rejection sampling preserves the target's output distribution: the target verifies every token.

How it works, in plain words: docs/HOW_IT_WORKS.md · Version history: EVOLUTION.md · CHANGELOG.md · FAQ

Data

Training prompts were drawn from public datasets (see CREDITS.md) plus synthetic agentic conversations. Every training target was generated by the target model itself.

Support the work

AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development:

Become a member on Patreon → patreon.com/cw/AeonForge7/membership

Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Supporters also get early access to new releases, such as the AEON DFlash2 drafter.

Attribution & sharing

You may share or build on this model under Apache-2.0. Keep the LICENSE and NOTICE files, and credit AEON-7 and z-lab / Inco AI.

• AEON-7: fine-tuning for the AEON Ultimate target, NVFP4 (W4A16) packing, serving config.
• Inco AI / z-lab: DFlash 2 and the original Qwen3.8-27B-DFlash2 (blog, code).
• Qwen team, Alibaba: Qwen3.8-27B, the base of the target model.
• Full acknowledgements, including vLLM, SpecForge, NVIDIA Model Optimizer, Marlin, FlashInfer, every training dataset and every benchmark dataset, with licenses: CREDITS.md.

@misc{aeon7_2026_dflash2_qwen38_27b,
  title  = {AEON DFlash 2 for Qwen3.8-27B AEON Ultimate},
  author = {AEON-7},
  year   = {2026},
  note   = {Version 1.0 (lk4, S5 step 753). Fine-tuned from z-lab/Qwen3.8-27B-DFlash2},
  url    = {https://huggingface.co/AEON-7/AEON-DFlash2-Qwen3.8-27B}
}

@misc{inco2026dflash2,
  title  = {DFlash 2: Keep Drafting Parallel},
  author = {Inco AI},
  year   = {2026},
  month  = {August},
  url    = {https://inco.ai/blog/dflash2/}
}

@inproceedings{chen2026dflash,
  title     = {DFlash: Block Diffusion for Flash Speculative Decoding},
  author    = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  booktitle = {ICML},
  year      = {2026}
}

License

Apache-2.0. A modified (fine-tuned) version of z-lab / incoai Qwen3.8-27B-DFlash2, also Apache-2.0. See the LICENSE and NOTICE files.