Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-MXFP4-MXFP6-ROCm

The AEON Ultimate uncensored 27B, quantized to OCP Microscaling (MX) formats for AMD GPUs. The MLP runs in MXFP4, the Gated DeltaNet projections in MXFP6, and everything that carries the model's skills stays in BF16. The checkpoint is 23.7 GB on disk and loads as 20.76 GiB of text-only weights (measured on one R9700). Two 32 GB Radeon AI PRO R9700 cards are estimated to hold one full 262,144-token sequence at a time (text-only, BF16 KV cache, after graph and activation reservations; estimated from measured overheads, not yet run at 262K; what that means): on the stock path (AEON_MXFP6_ONCE=0), and with room to spare in the community-tested optimized runtime's measured 64K pool. The faster default recipes use 131,072 (A-MTP, A-plain) or 65,536 (A, DFlash2). One R9700 runs it eager at 16K context (measured, 0.86 tok/s; 32K estimated to fit). MXFP4 is also the format Instinct MI350 / MI355 execute natively. The Hub's file panel labels this repo "8-bit" only because the MX weights are stored packed in U8 tensors; the real mix is MXFP4 MLP + MXFP6 Gated DeltaNet projections + BF16 for the rest, about 6.8 bits per weight on average: a 23.7 GB download that needs at least 32 GB of total VRAM.

Recommended on 2x R9700: vLLM with tensor parallelism across both cards, this checkpoint unchanged, and the bundled AEON DFlash2 drafter for lossless speculative decoding (Recipe A).

[!WARNING] Experimental: early access. The first community results on 2x R9700 are in (Measured on 2x R9700): stock emulation decodes at about 1.7 tok/s, and a community-tested optimized runtime reached 49.8 tok/s single-request with the same W4A4/W6A6 quantization (unchanged checkpoint, identical 128-token greedy text); its integration into this kit is in progress. The kit's own patched recipes and the full quality score are still pending. Requests are reviewed by hand, usually within a few hours; include the community access word in the form if you were given one.

AEON DFlash2 drafter: an earlier stage is bundled in dflash2/ (see Recipe A). The latest v1.0 is on Hugging Face (its bf16/ copy is the one to use on ROCm), with initial gated access for Patreon supporters: early-access post · membership.

Measured on 2x R9700 by Todd (community tester)

The first AMD results for this checkpoint come from Todd (community tester): 2x Radeon AI PRO R9700 (gfx1201) plus a gfx1036 integrated GPU kept out of the container, host ROCm 7.2.4, image vllm/vllm-openai-rocm@sha256:749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f (the v0.31.0 digest the scripts now pin). He used the published weights from an earlier revision of this repo, before AGENTS.md and scripts/ were uploaded, so these runs used the card's docker run commands and his own runtime builds, not this kit's MXFP6 dequant-once patch. Where they overlap, these measurements replace our estimates. Speeds are single-request decode, greedy, thinking off, 128 output tokens unless noted: diagnostics, not a quality benchmark.

What this shows:

• Stock emulation is slow, and TP=2 is not the reason. About 1.7 tok/s at TP=2 and 0.86 at TP=1. After the MXFP6 fix, a GPU trace of the TP=2 runtime put BF16 GEMM at 71.9% of kernel time, the fused MXFP6 dequant at 14.3% and RCCL at 7.3%.
• Re-dequantizing MXFP6 on every forward pass is the main bottleneck (about 8.7 ms to unpack one layer's three projections, against about 0.2 ms for their BF16 GEMMs). This kit's MXFP6 dequant-once patch is the "MXFP6 at load" step Todd recommends: weights dequantized once at load, activation MXFP6 quantize-dequantize kept. It is on by default in A, A-MTP and A-plain (AEON_MXFP6_ONCE=1; log line AEON_MXFP6_DEQUANT_AT_LOAD=1: MXFP6 weights dequantized once at load). In his runtime that step took the packed-MLP setup from 38.1 to 49.8 tok/s and cost +3.19 GiB per card; the kit's patch now has the same net cost (it releases the packed copy). The kit's own patch file has not run on AMD yet.
• A community-tested optimized runtime reached 49.8 tok/s single-request on 2x R9700 (packed MXFP4 W4A4 on gfx1201 + MXFP6 at load, MTP 3, 64K context), with this checkpoint unchanged and the same W4A4/W6A6 quantization: all 144 MXFP6 tensors (5.54 billion values) matched stock dequantization bit for bit, packed MXFP4 activation quantization and the tested MLP outputs matched the stock path, the fixed 128-token greedy text was identical across every variant, arithmetic and tool-call parsing passed, and a 7,040-token retrieval prompt returned the right marker. Its integration into this kit is in progress (see Community-tested optimized runtime); until it lands, the recipes here don't include its packed W4A4 MLP or its fused kernel.
• Memory. The measured weight, graph and activation reservations now drive every memory figure on this card (Memory). The defaults that failed were changed to the settings that worked, and the other recipes were re-sized to the low end of the measured overheads.
• Not yet measured: the kit's patched recipes (A, A-MTP, A-plain), DFlash2 on RDNA 4, and the full AEON quality score of the optimized runtime (in progress).

Context length: what 262K means

• --max-model-len is per sequence (prompt plus output), not the total KV cache. All running sequences share one pool; the startup line Maximum concurrency for N tokens per request: Cx says how many full-length sequences fit at once.
• Two 32 GB R9700s are estimated to hold one full 262,144-token sequence at a time (text-only, BF16 KV cache, after graph and activation reservations; estimated from measured overheads, not yet run at 262K) on stock emulation: AEON_MXFP6_ONCE=0 AEON_MAX_MODEL_LEN=262144 AEON_MAX_NUM_SEQS=2 bash "$AEON_DIR/scripts/serve_2x_r9700.sh" (same output; estimated pool 12–14 GiB per card, about 1.5–1.7 full-length sequences; ~1.7 tok/s measured on that path at 131,072). The community-tested optimized runtime should have room for it too (11.83 GiB of pool per card measured at 64K with MTP 3, against an estimated ~8.9 GiB for one 262K MTP sequence; not run at 262K). With this kit's MXFP6 patch it is thin (estimated 8.9–10.6 GiB of pool against ~8.2 GiB for one sequence: check the log), and with DFlash2 it doesn't fit (~10.8 GiB per card for one sequence).
• The faster default recipes use less: 131,072 (A-MTP, A-plain) or 65,536 (A, DFlash2), so several requests fit at once.
• MTP adds 0.40 GiB of weights per card at TP=2, a 17th attention layer (+6% KV per token) and 3 Gated DeltaNet state pages per running sequence. Vision is excluded (--language-model-only); it adds about 0.86 GiB of weights.
• Measured (optimized runtime, 65,536 configured): 308,086 KV tokens in total = 4.70 full-length sequences, 11.83 GiB of pool per card.

At a glance

Why it's great

• Quality first. Only two parts of the model are quantized: the MLP (MXFP4, W4A4) and the Gated DeltaNet projections (MXFP6, W6A6). Full attention, the recurrence, embeddings, lm_head, the vision tower and the MTP head all stay in BF16. That is about 6.8 bits per weight: 23.7 GB, against 55.6 GB for the master.
• Full context on two 32 GB cards. 2x Radeon AI PRO R9700 are estimated to hold one full 262,144-token sequence at a time (from measured overheads; not yet run at 262K; what that means); the default recipes trade some of it for concurrency. Instinct MI350X / MI355X can also run the MXFP4 MLP natively, as an opt-in.
• Speculative decoding ships with it. The repo includes the AEON DFlash2 drafter (dflash2/, BF16) alongside the model's own MTP head, and both are lossless. In the reference run, DFlash2's mean acceptance length was 3.73 at T=0.6, and it ran 2.7x faster than plain decoding on the same 4 prompts (details).
• Turnkey across the ROCm lineup. There is a launch script per setup in scripts/, and detect_setup.py picks the right one for your GPUs. The MXFP6 dequant-once patch gives the same output as stock vLLM with a much faster forward pass; it is the "MXFP6 at load" step behind the community-measured speedup.
• AEON Ultimate, uncensored. It is the full 27B hybrid (thinking, tool calling, and vision on Instinct and RDNA 3: see Vision), and it writes what the base model refuses. Read User responsibility.

Target systems (first community results on 2x and 1x R9700: Measured; everything else untested)

Recommended configuration

• 2x R9700: Recipe A. Launch it with bash "$AEON_DIR/scripts/serve_2x_r9700_dflash2.sh". It runs:

  • vLLM v0.31.0 (ROCm image) at TP=2, with the bundled drafter at K=9 and the lattice [[1,2,9],[3,8,7],[9,12,6],[13,16,4]];
  • TRITON_ATTN, including inside --speculative-config, with VLLM_USE_V2_MODEL_RUNNER=1 and prefix caching off.

  The estimate is about 20 tok/s single-stream (calibrated against the community per-component measurements), ±50% until measured. Measured so far on 2x R9700: stock emulation ~1.7 tok/s; the community-tested optimized runtime 49.8 tok/s (Measured).

• Before any long benchmark: python3 "$AEON_DIR/scripts/sanity_decode.py" --container aeon-mxfp4 (2–10 minutes). It stops with exit code 5 below 5 tok/s single-stream (3 for the exact-baseline A-plain arm inside run_matrix.sh and preflight_smoke.sh), so a slow server doesn't turn a quality run into days of timeouts.

• Long multi-turn agent loops: A-MTP keeps prefix caching on.

• Exact baseline, or the most concurrent requests: A-plain.

• TP=2 hangs: fallback to test, Z.

• Any other setup: run python3 "$AEON_DIR/scripts/detect_setup.py".

Using the bundled DFlash2 drafter with other engines

dflash2/ is a standard DFlash2 checkpoint. It has the same 81 tensors (names, shapes and dtypes) as z-lab/Qwen3.8-27B-DFlash2. Its config differs only in block_size 10, causal: false and draft_vocab_size. By format, any engine that runs z-lab's drafter loads it if you point its drafter path at dflash2/ (only vLLM has been run with an AEON target, and not yet on AMD). Set the draft width to 9: num_speculative_tokens 9 in vLLM, --speculative-num-draft-tokens 10 in SGLang (the block size), or --spec-draft-n-max 9 in llama.cpp. For this mixed checkpoint, vLLM is the only engine that loads the target unchanged (see TensorFold). Full guide: dflash2/README.md.

What this is

• Base model: AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 is AEON-7's uncensored build of Qwen/Qwen3.8-27B. It is a hybrid of 48 Gated DeltaNet (linear-attention) layers and 16 full-attention layers, with a vision tower, an MTP head and a native context of 262,144 tokens.

• Format: OCP Microscaling. Every block of 32 values shares one power-of-two (E8M0) scale. We used AMD Quark 0.12.post1 and exported real packed weights, not fake-quant.

  • MLP (gate_proj, up_proj, down_proj) is MXFP4, W4A4: FP4 E2M1 weights, with activations quantized to MXFP4 on the fly.
  • Gated DeltaNet projections (in_proj_qkv, in_proj_z, out_proj) are MXFP6 E2M3, W6A6.
  • The rest stays BF16: full attention, embeddings, lm_head, the vision tower, the MTP head, the GDN recurrence (conv1d, in_proj_a/b, A_log, dt_bias) and the norms. Precision lattice (Quark ROCm): MLP MXFP4 - GDN writers MXFP6 - attention / vision / MTP / embed / lm_head / GDN recurrence BF16.

• Unchanged from the BF16 master: the tokenizer, chat template and vision processor. Thinking is on by default.

Why MX on AMD? MX is the open block-scaled format that AMD's newest accelerators run in hardware. On Radeon the gain is footprint: the BF16 master is 55.6 GB, while this build is 23.7 GB. The precision is mixed on purpose. The MLP and GDN projections take 4-bit and 6-bit weights, and the attention path, the recurrence, the embeddings and the vision tower stay in full BF16.

Target hardware

Every AMD GPU configuration that runs ROCm compute and can hold this checkpoint has its own labelled quickstart below. The featured setup is 2x Radeon AI PRO R9700 (Recipe A and its fallbacks).

• Native MXFP4: Instinct MI350X / MI355X (gfx950) can run the MLP on AITER's W4A4 GEMM. That path uses the same MX grid but is not bit-identical to the other recipes, so Recipe J defaults to the exact emulated path and offers native MXFP4 as a labelled opt-in. MXFP6 is emulated on every GPU, because vLLM has no MXFP6 kernel yet.
• Emulated: everything else, including Radeon RDNA 4 (R9700, RX 9070 / 9060 XT), RDNA 3 (RX 7900 / 7800 / 7700, W7900, W7800, W7700, V710), Strix Halo and Instinct MI300X / MI325X / MI300A / MI250 / MI210. vLLM keeps the packed MX weights in memory, dequantizes each layer to BF16 on every forward pass, quantizes and dequantizes the activations to the same MX grid, and runs the matmul in BF16. The numerics match W4A4 and W6A6 and memory stays near the packed size, but decoding is slower than a native kernel would be (vLLM 0.29 and 0.31 source: QuarkOCP_MX, EmulationMxfp4LinearKernel, EmulationMxfp6LinearKernel; native MX needs supports_mx(), which is true only on gfx95x and gfx1250). Wherever memory allows, the recipes remove most of that cost without changing a bit of the output: the MXFP6 weights are dequantized once at load (a small vLLM patch in scripts/patches/) and, on the largest GPUs, the MXFP4 MLP is kept in BF16 after load (VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD).

Quickstart

Deploying with an AI agent (Claude Code, Codex, Cursor)? Point it at AGENTS.md. It has the preflight, every recipe with exact commands, verification steps and an exhaustive troubleshooting table.

Testing before an event? Run bash "$AEON_DIR/scripts/preflight_smoke.sh" the day before. It checks the host, starts Recipe A (falling back to A-MTP, A-plain, then the fallback to test, Z; a TP=2 start that hangs goes straight to Z), runs the decode sanity test (scripts/sanity_decode.py), smoke-tests it in a few minutes and bundles the results to send back. mkdir -p ~/aeon-test && cd ~/aeon-test && bash "$AEON_DIR/scripts/run_matrix.sh" then compares plain, MTP and DFlash2 at 1–16 concurrent requests and checks TP=2 against a one-GPU reference.

Step 0: Which setup do you have? (1 minute)

Run this on the Linux host. It reads the amdgpu driver directly, so it needs no ROCm tools, no Docker and no root:

# one line per AMD GPU: gfx target, VRAM, render node, PCI name
for p in /sys/class/kfd/kfd/topology/nodes/*/properties; do
  v=$(awk '$1=="gfx_target_version"{print $2}' "$p"); [ "${v:-0}" -gt 0 ] || continue
  r=$(awk '$1=="drm_render_minor"{print $2}' "$p"); d=/sys/class/drm/renderD$r/device
  printf 'gfx%d%x%x  %5.1f GiB  /dev/dri/renderD%s  %s\n' $((v/10000)) $((v/100%100)) $((v%100)) \
    "$(awk '{print $1/2^30}' "$d/mem_info_vram_total")" "$r" \
    "$(lspci -s "$(basename "$(readlink -f "$d")")" 2>/dev/null | cut -d' ' -f2-)"
done

With ROCm tools on the host, amd-smi static --asic --vram or rocm-smi --showproductname --showmeminfo vram show the same thing. After the download, python3 "$AEON_DIR/scripts/detect_setup.py" does this lookup and prints the matching recipe and launch script, and bash "$AEON_DIR/scripts/preflight.sh" checks the rest of the host (driver, groups, PCIe slots, IOMMU, Resizable BAR, disk, ports).

Match the output to the table below. Count only the discrete GPUs.

• Ignore an integrated GPU (gfx1036, gfx1035, gfx1103, gfx1150 and similar, with a few GiB of VRAM), unless it gets passed into the container. vLLM reads the GPU architecture from the first GPU that amdsmi lists, and HIP_VISIBLE_DEVICES doesn't change that. If the iGPU comes first, the RDNA 4 paths switch off.
• To hide it, either disable the iGPU in the BIOS, or pass only the discrete GPUs' nodes into the container: replace --device /dev/dri with --device /dev/dri/renderD128 --device /dev/dri/card1 ... for each discrete GPU. The launch scripts do this with AEON_DEVICES=auto (switched on by themselves when detect_setup.py sees an iGPU), and they refuse to start if vLLM would see the wrong architecture or would put a tensor-parallel rank on an iGPU (AEON_ALLOW_IGPU=1 overrides, for testing only). This is community-tested: a 2x R9700 host with a gfx1036 iGPU served correctly with only the two R9700s' renderD* / card* nodes passed in.

Pick your system

Status: Tested = run with this checkpoint on that hardware by a community tester, with the pinned image (Measured). Partly tested = some configurations on that hardware have run, the recipe's own defaults haven't. Untested = not yet run on that hardware. "Expected to work" means the vLLM code path for that architecture and GPU count is confirmed from source, and other Qwen3.x checkpoints are reported running in vLLM on the same GPU family with the same settings. Not supported = it doesn't fit, or vLLM can't run there.

MX path: once = MXFP6 dequantized once at load; at load = MXFP4 MLP also kept in BF16 after load; stock = both dequantized on every forward pass (no room for the BF16 copies). All three give identical output; they differ in speed and memory. eager = --enforce-eager by default: on these cards HIP-graph capture (2.0–2.5 GiB per GPU, measured on R9700) would leave little or no KV cache (measured). Eager vs graphs has not been A/B-measured; TP=1 eager at 0.86 tok/s against TP=2 with graphs at about 1.7 is consistent with eager costing little on the stock path.

Common setup (every recipe)

You need Linux on bare metal (not a VM with GPU passthrough for multi-GPU recipes: RCCL is reported to hang there), a recent amdgpu driver for your GPU, Docker, and about 100 GB of free disk (model 23.7 GB, drafter 3.85 GB, the image's 11.5 GB download and more once unpacked, kernel caches; about 12 GB more per extra image tag).

Every recipe uses the official vLLM ROCm image vllm/vllm-openai-rocm:v0.31.0, pinned by digest (ROCm 7.2.3; built for gfx90a, gfx942, gfx950, gfx1100, gfx1101, gfx1150, gfx1151, gfx1200 and gfx1201; includes amd-quark). The scripts default to vllm/vllm-openai-rocm@sha256:749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f, the image the community 2x R9700 test ran on, print the digest they actually start, and record vLLM, Quark, torch, HIP, ROCm and RCCL versions from inside the image in image_check_<recipe>.txt.

• Don't use v0.28, v0.29 or v0.30, except through the AEON_STRIP_ALGO_CONFIG=1 escape hatch (AGENTS.md section 5.6). They crash while loading this checkpoint with AttributeError: 'dict' object has no attribute 'endswith', because their Quark loader can't handle the list-valued algo_config in config.json. v0.31.0 fixes it.
• Don't use the nightly / rocm100 images on Radeon. They ship no gfx1201 kernels (ROCm/aiter#5229).
• The digest is pinned. As of 2026-10-04 vllm/vllm-openai-rocm:v0.31.0 is sha256:749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f, and that is what the scripts and the docker run commands below use (AEON_IMAGE). A tag can be re-pushed; a digest can't. If you start from the tag instead, the launcher warns when its digest differs.

# 1) After you accept the access terms on the model page (log in with your own account and token)
python3 -m venv ~/.venvs/hf && ~/.venvs/hf/bin/pip install -U huggingface_hub && export PATH="$HOME/.venvs/hf/bin:$PATH"   # Ubuntu may need: sudo apt install python3-venv
hf auth login
hf download AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-MXFP4-MXFP6-ROCm \
  --local-dir ~/models/aeon-mxfp4-rocm
#   the download includes dflash2/ (3.85 GB): the DFlash2 drafter that Recipes A, K and Q use. Skip it with
#   --exclude "dflash2/*" only if you will not use DFlash2.

# 2) Variables the commands below use (re-run in every new terminal)
export AEON_DIR="$HOME/models/aeon-mxfp4-rocm"
export AEON_CACHE="$HOME/.cache/aeon-mxfp4-rocm"     # compiled kernels persist here
export AEON_IMAGE="vllm/vllm-openai-rocm@sha256:749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f"   # pinned v0.31.0 (community-tested)
export RENDER_GID="$(getent group render | cut -d: -f3)"; [ -n "$RENDER_GID" ] || RENDER_GID=video
mkdir -p "$AEON_CACHE" ~/aeon-test && cd ~/aeon-test    # logs and results land in the current folder

# 3) Check the host, and see which recipe fits
bash "$AEON_DIR/scripts/preflight.sh"
python3 "$AEON_DIR/scripts/detect_setup.py"

Each recipe below has a one-line launch script (in scripts/, shipped with the model) and the plain docker run it executes. The scripts also check which GPUs and which vLLM version the container sees before starting, refuse known-bad combinations, wait for the server, and save the startup log. Every value can be changed through an environment variable, for example AEON_MAX_MODEL_LEN=65536 AEON_MAX_NUM_SEQS=8 bash "$AEON_DIR/scripts/serve_2x_r9700.sh"; AEON_DRY_RUN=1 prints the command without starting anything. The header of each script lists its target system, status and memory estimate. Check every start with Verify.

Applies to every recipe:

• Don't pass --quantization. vLLM reads the Quark config from config.json.
• The first start is slow (allow 10–30 minutes). vLLM loads the weights, then Triton compiles and autotunes the Gated DeltaNet and attention kernels, and Quark JIT-compiles its MX emulation kernel with hipcc. The cache mount keeps all of that for later starts.
• PYTORCH_ROCM_ARCH makes Quark compile for your one GPU target instead of the nine the image lists. The scripts set it to the architecture the container detects. In a plain docker run, use your own target (for example gfx1200 for an RX 9060 XT, gfx1101 for an RX 7800 XT, W7700 or V710): a kernel built for a sibling target fails with invalid device function.
• BF16 KV cache everywhere (--kv-cache-dtype auto). The checkpoint ships no calibrated KV scales, so FP8 KV would run with a scale of 1.0 and change the outputs. FP8 KV is an opt-in (AEON_KV_DTYPE=fp8) for 16 GB cards only.
• VLLM_ROCM_USE_AITER=0 on every recipe. On Radeon, AITER's unified attention overflows RDNA 4's 64 KB LDS. On MI350/MI355 the opt-in native MXFP4 GEMM is picked without it.
• --attention-backend TRITON_ATTN everywhere, and inside --speculative-config too. The draft model doesn't inherit the target's backend; vLLM's automatic pick on ROCm (ROCM_ATTN) collapses speculative acceptance under concurrency.
• MXFP6 dequant-once patch wherever it fits (AEON_MXFP6_ONCE=1; Recipes A, A-MTP, A-plain, D, E, F, H, H-128, I, J, K, N, O, Q): the same output as stock vLLM, an estimated 3–5x faster forward pass (about 3.5x on 2x R9700: about 590 ms to about 170 ms, from the community measurements). This is the "MXFP6 at load" configuration Todd (community tester) recommends: weights dequantized once, activation MXFP6 quantize-dequantize kept; his 2x R9700 runtime measured 14.1 GiB of weights per card with it (MXFP4 MLP packed) (Measured). The patch releases the packed MXFP6 copy after the one-time dequant, so it costs about 6.3 GiB net in total, divided by the number of GPUs (about 3.1 GiB per GPU at TP=2; it was 10.3 / 5.2 while the packed copy stayed allocated, which AEON_MXFP6_KEEP_PACKED=1 restores for an A/B). MXFP4 dequant-at-load (AEON_DEQUANT_AT_LOAD=1; H-128, I, J, K, O, Q): bit-identical, +23.8 GiB in total, divided by the number of GPUs (+11.9 per GPU measured at TP=2).
• --override-generation-config is required. The shipped generation_config.json says temperature 1.0; the override sets the model card's 0.6 / 0.95 / 20.
• Multi-GPU Radeon uses NCCL_PROTO=Simple (from the community R9700 TP=2 / TP=4 guide) and NCCL_P2P_DISABLE=1 (kept for robustness against hipIpc errors; that guide found it unnecessary). The scripts set both whenever TP > 1 on a Radeon or Ryzen GPU. The same guide boots with amd_iommu=on iommu=pt pcie_aspm=off.
• One compile cache per weight layout. vLLM's torch.compile cache key doesn't see the mounted patch or the MX switches, so artifacts compiled for one layout (for example a BF16-resident MLP) can be reused for another and fail at startup with shape / stride errors (reported by Todd). The scripts give every layout its own VLLM_CACHE_ROOT under $AEON_CACHE/vllm/ (AEON_CACHE_KEY, derived from the GPU, TP, MX path, speculation, image and patch files) and keep Triton's content-addressed cache shared. The docker run commands below set one VLLM_CACHE_ROOT per recipe; give a hand-edited variant (another MX path, MTP on or off) its own.
• Ready when the log prints Application startup complete. (docker logs -f aeon-mxfp4). Then run the decode sanity test before anything long: python3 "$AEON_DIR/scripts/sanity_decode.py" --container aeon-mxfp4 (first-request time, warmed tok/s at 128 tokens and on varied prompts up to 384 output tokens, TTFT, KV capacity; exit code 5 and STOP below 5 tok/s). Stop the server with docker rm -f aeon-mxfp4.

2x R9700: pick a recipe

All four recipes serve this checkpoint byte for byte with its own numerics: W4A4 MXFP4 MLP, W6A6 MXFP6 Gated DeltaNet projections, BF16 everywhere else, BF16 KV cache. A, A-MTP and A-plain differ only in speculative decoding, which is lossless in distribution (vLLM's standard rejection sampler keeps the target's output distribution). The kit's patched A, A-MTP and A-plain haven't run on RDNA 4 yet (the stock path, Recipe B and A-Max have: Measured). The acceptance figures are measured τ (mean acceptance length) for this exact checkpoint, drafter and speculative config, and the R9700 speeds are estimates.

• Which one. Start with A. It has the most tokens per forward pass, and under emulation every pass is expensive, so that is where speed comes from. For long multi-turn agent loops (each turn resends a growing conversation), also try A-MTP: DFlash2 needs prefix caching off on this hybrid model in vLLM 0.31 (see Recipe A), so every turn re-prefills the whole conversation, while A-MTP reuses the cached prefix. If many long-context agents share one server, A-plain and A-MTP hold more of them at once (table). scripts/run_matrix.sh measures all three on your machine (see Tester matrix).
• Fall back in this order: A → A-MTP → A-plain → A-plain on the v0.28.0 image (only after a vLLM-level error, not an RCCL hang: AGENTS.md section 5.6) → Z. A-plain is also the reference the other two are compared against.
• The MXFP6 dequant-once patch (scripts/patches/, on in A, A-MTP and A-plain). Stock vLLM re-dequantizes the MXFP6 Gated DeltaNet weights on every forward pass. That one step was measured at about 2.1 s per pass on the reference run, 48x the same layers in BF16, and the community 2x R9700 test confirmed it as the main bottleneck: stock emulation decoded at about 1.7 tok/s (Measured). The patch calls the same dequant function once at load, reuses the result and releases the packed copy: output identical to stock (checked with torch.equal), at a net cost of about 3.1 GiB per GPU. It is the "MXFP6 at load" step Todd (community tester) recommends; in his runtime that step alone took 38.1 to 49.8 tok/s. The launcher mounts it read-only over the one vLLM file it replaces, only after checking that file is byte-identical to the v0.31.0 original. AEON_MXFP6_ONCE=0 turns it off.
• How the speed estimates are made. Measured (one request, T=0.6): on 4 identical prompts plain 8.1, MTP 13.5 and DFlash2 22.0 tok/s (2.7x); on 13 identical prompts MTP τ 2.76 vs DFlash2 τ 4.20; DFlash2 over all 26 prompts τ 3.73, 23.2 tok/s. Calibrated on 2x R9700 with the community per-component measurements: stock TP=2 decodes at about 1.7 tok/s (about 590 ms per token), of which unpacking MXFP6 takes about 8.7 ms per Gated DeltaNet layer, about 420 ms over 48 layers; the patch removes that and leaves about 170 ms per forward pass (MXFP4 emulation, the BF16 layers, activation quantize-dequantize, all-reduce), about 6 tok/s plain. Speedup = mean acceptance length ÷ the relative cost of a drafting-plus-verify step (about 1.14 for DFlash2, 1.2 for MTP). A bandwidth-only model gave about 94 ms per pass, about 1.7x optimistic. Treat every unmeasured R9700 tok/s figure as ±50%; the patched recipes stay far below the community-tested optimized runtime's 49.8 tok/s, which also packs the MLP as W4A4.
• Concurrency. The weight traffic is per forward pass, not per sequence, so aggregate throughput rises with concurrent requests. DFlash2 delivered 4.9x its single-stream throughput at 8 concurrent requests (113 vs 23 tok/s, measured). c=16 is unmeasured. How many requests fit at once is set by the KV and Gated DeltaNet state pool (last column; the startup log's GPU KV cache size is the real number).

Verify (every recipe)

Run this after any launch, in the same terminal. For Recipe Z use NAME=aeon-mxfp4-0 (the proxy is on port 8080).

NAME=aeon-mxfp4; PORT=8000
AUTH=(); [ -n "${VLLM_API_KEY:-}" ] && AUTH=(-H "Authorization: Bearer $VLLM_API_KEY")   # an array, so the header stays one argument
# 1) wait until ready (first start: 10-30 minutes); stops and prints the log if the container died
until curl -sf "${AUTH[@]}" http://127.0.0.1:$PORT/health >/dev/null; do
  docker inspect -f '{{.State.Running}}' $NAME 2>/dev/null | grep -q true || { docker logs --tail 80 $NAME; break; }
  sleep 15
done
# 2) smoke: expect READY
curl -s "${AUTH[@]}" http://127.0.0.1:$PORT/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "aeon", "max_tokens": 64,
  "messages": [{"role": "user", "content": "Reply with exactly: READY"}],
  "chat_template_kwargs": {"enable_thinking": false}}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
# 3) speculative recipes (A, A-MTP, K, Q, any AEON_MTP / AEON_DFLASH run), after some traffic:
curl -s "${AUTH[@]}" http://127.0.0.1:$PORT/metrics | python3 -c '
import re, sys
v = {}
for line in sys.stdin:
    m = re.match(r"^(vllm:spec_decode_num_(drafts|draft_tokens|accepted_tokens))(?:_total)?(?:\{[^}]*\})? ([0-9.e+]+)$", line)
    if m: v[m.group(2)] = v.get(m.group(2), 0) + float(m.group(3))
d, a = v.get("drafts", 0), v.get("accepted_tokens", 0)
print("mean acceptance length", round(1 + a / d, 2) if d else "no drafts yet")'

Healthy: DFlash2 about 3 or more at one request, MTP about 2 or more. About 1 with several requests in flight means the draft is not on TRITON_ATTN. AGENTS.md section 6 has the full checklist (log lines, tool calls, reasoning field).

Then the decode sanity test (2–10 minutes; --container also reads the KV capacity and compile times from the log):

python3 "$AEON_DIR/scripts/sanity_decode.py" --base-url http://127.0.0.1:$PORT/v1 --container $NAME --label my_run
# STOP + exit code 5 = below 5 tok/s single-stream (AEON_SANITY_MIN_TOK_S): fix the recipe before any long benchmark

Recipe A: 2x R9700 with the AEON DFlash2 drafter (primary)

Target: 2x AMD Radeon AI PRO R9700 · VRAM: 2 × 32 GB · Arch: gfx1201 (RDNA 4) · MX path: emulated, MXFP6 dequantized once at load; drafter in BF16 · Context: 65,536 (131,072 opt-in) · Status: Untested on RDNA 4 (measured τ 3.73 at T=0.6 for this checkpoint, drafter and config) · Script: scripts/serve_2x_r9700_dflash2.sh · Needs: the dflash2/ folder (AEON-DFlash2-Qwen3.8-27B-BF16, 3.85 GB)

bash "$AEON_DIR/scripts/serve_2x_r9700_dflash2.sh"        # AEON_DFLASH=<dir> if the drafter lives elsewhere

Plain docker run for Recipe A

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/dflash2":/draft:ro \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1201 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_r9700_dflash2 \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 -e VLLM_USE_V2_MODEL_RUNNER=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 2 --max-model-len 65536 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 8192 \
  --attention-backend TRITON_ATTN --no-enable-prefix-caching --language-model-only \
  --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":9,"num_speculative_tokens_per_batch_size":[[1,2,9],[3,8,7],[9,12,6],[13,16,4]],"attention_backend":"TRITON_ATTN","draft_sample_method":"probabilistic","rejection_sample_method":"standard"}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

The patch path assumes vLLM lives at /usr/local/lib/python3.12/dist-packages/vllm in the image, as it does in v0.31.0. Check with docker run --rm --entrypoint python3 "$AEON_IMAGE" -c 'import vllm,os; print(os.path.dirname(vllm.__file__))'. The script finds the path itself and also checks the file's hash.

Why these settings:

• The drafter. AEON-DFlash2-Qwen3.8-27B-BF16 is a DFlash 2 block-diffusion drafter fine-tuned for this model from z-lab's Qwen3.8-27B drafter (Apache-2.0). It reads hidden states from five target layers and proposes 9 tokens per step; the target checks them in one forward pass. It is BF16, so it runs on ROCm as is.
• "num_speculative_tokens":9 matches the drafter's training block of 10 (vLLM drafts 1 + K positions). The per-batch-size lattice [[1,2,9],[3,8,7],[9,12,6],[13,16,4]] lowers K as more requests share a step; it is the schedule the acceptance was measured with. AEON_DFLASH_LATTICE= (empty) keeps K=9 at every batch size.
• "attention_backend":"TRITON_ATTN" inside --speculative-config. The draft model doesn't inherit --attention-backend. vLLM's automatic pick on ROCm is ROCM_ATTN, which collapses DFlash2 acceptance once more than one request is in flight (vllm#53323: acceptance length 1.47 instead of 4.99 at 4 requests).
• VLLM_USE_V2_MODEL_RUNNER=1. DFlash2 exists only in vLLM's V2 model runner. With the variable set to 0, vLLM quietly loads the drafter as DFlash 1.
• "draft_sample_method":"probabilistic", "rejection_sample_method":"standard". Both are lossless. Probabilistic drafting is what was measured; AEON_DFLASH_SAMPLE=greedy is the alternative. Never use "synthetic": it is not lossless.
• --no-enable-prefix-caching. In vLLM 0.29–0.31, a hybrid Gated DeltaNet model with a DFlash2 drafter crashes or drops to 0% acceptance after a prefix-cache hit (vllm#58894, fix proposed in vllm#55601). Every request therefore prefills its whole prompt. For one-shot work that costs nothing extra; for long multi-turn loops, compare against A-MTP. AEON_DFLASH_PREFIX_FIX=1 mounts the proposed one-line fix and keeps prefix caching on: experimental and untested.
• --dtype bfloat16 avoids a drafter dtype mismatch reported on ROCm (vllm#42588) and fp16's 0% acceptance (vllm#55250).
• TP=2, NCCL settings, --language-model-only, BF16 KV: as in Recipe A-plain.
• Memory (estimate) per GPU: 10.51 GiB of weights (measured in the community test), about 3.1 GiB net for the MXFP6 BF16 copy and 1.8 GiB for the drafter (it shares the target's embeddings and lm_head): about 15.4 GiB. Graphs about 2.3–3.1 GiB and activations about 2.4–3.6 GiB at 8,192 batched tokens leave a pool of about 6.5–8.6 GiB for KV cache and Gated DeltaNet state. vLLM pads the target's 16 attention layers to 20 to group them with the drafter's 5 layers (the log warns Add 4 padding layers, may waste at most 25.00% KV cache memory: expected), so KV costs about 40 KiB per token per GPU, plus Gated DeltaNet state and drafter KV per running request. One 65,536-token request needs about 3.3 GiB (Maximum concurrency for 65,536 tokens per request about 2.0–2.6x); a 16K-token request about 1.45 GiB (about 4–6 at once). Expect GPU KV cache size around 130–170K tokens. More requests wait in the queue; with many long-context agents, AEON_MAX_NUM_SEQS=8 avoids preemption churn, or use A-plain / A-MTP, which hold more.
• Why 65,536 and not 131,072. One 131,072-token DFlash2 request needs about 5.8 GiB per GPU, which doesn't fit reliably at the low end of the measured overheads (two community-measured start failures, Recipe B and A-Max, both sat at that low end). AEON_MAX_MODEL_LEN=131072 is an opt-in: keep it only if the 65,536 start logs Maximum concurrency of at least 2.0. If profiling runs out of memory, AEON_MAX_BATCHED=4096 frees an estimated 0.4–0.8 GiB per GPU.
• Pass check. The log shows Resolved architecture: DFlash2DraftModel, MXFP6 weights dequantized once at load, and under load SpecDecoding metrics: Mean acceptance length: ... of about 3 or more. About 1 with several requests in flight means the draft is not on TRITON_ATTN.
• If it fails to load, or acceptance stays near 1, go to A-MTP and send us the log.

Recipe A-MTP: 2x R9700 with MTP (alternative)

Target: 2x AMD Radeon AI PRO R9700 · VRAM: 2 × 32 GB · Arch: gfx1201 (RDNA 4) · MX path: emulated, MXFP6 dequantized once at load; the MTP head runs in BF16 · Status: Untested on RDNA 4 (measured τ 2.76 at T=0.6 for this checkpoint's MTP head) · Script: scripts/serve_2x_r9700_mtp.sh

bash "$AEON_DIR/scripts/serve_2x_r9700_mtp.sh"

Plain docker run for Recipe A-MTP

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1201 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_r9700_mtp \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 2 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 8192 \
  --attention-backend TRITON_ATTN --language-model-only \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• What it is. The model's own one-layer MTP head drafts 3 tokens; the main model checks them in one pass. config.json lists the MTP head's layers as excluded from quantization, so vLLM builds them in BF16 to match the weights on disk. Measured: mean acceptance length 2.76 at T=0.6 (DFlash2: 4.20 on the same 13 prompts).
• Why keep it. MTP keeps prefix caching on, so a new turn of a long conversation only prefills the new tokens. Under emulation prefill is expensive, so for long multi-turn agent loops A-MTP can finish turns sooner than A even though it decodes slower. The tester matrix measures both.
• Same as A-plain except speculation: identical environment and flags plus --speculative-config, so the matrix compares speculation only.
• Pass check. Resolved architecture: Qwen3_5MTP in the log (it follows Resolved architecture: Qwen3_5ForConditionalGeneration; vLLM 0.31 runs this model on its V2 runner, which does not print the V1 runner's Detected MTP model line), and under load SpecDecoding metrics: Mean acceptance length of about 2 or more.
• Memory (estimate): about 14.0 GiB of weights per GPU (10.51 measured + MTP 0.40 + MXFP6 net 3.1), pool about 8.5–10.2 GiB. The MTP layer adds a 17th attention layer (34 KiB per token per GPU), and each running request holds 2 + K Gated DeltaNet state pages (about 0.36 GiB per GPU at K=3). One 131,072-token request needs about 4.65 GiB (Maximum concurrency about 1.8–2.2x); about 9–11 concurrent 16K-token requests.
• Tuning: AEON_MTP=2 or AEON_MTP=4.

Recipe A-plain: 2x R9700 without speculation (exact baseline)

Target: 2x AMD Radeon AI PRO R9700 · VRAM: 2 × 32 GB · Arch: gfx1201 (RDNA 4) · MX path: emulated, MXFP6 dequantized once at load · Status: Untested (expected to work) · Script: scripts/serve_2x_r9700.sh

bash "$AEON_DIR/scripts/serve_2x_r9700.sh"

Plain docker run for Recipe A-plain

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1201 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_r9700 \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 2 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 8192 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Why these settings (shared by A and A-MTP):

• --tensor-parallel-size 2 splits every layer across both cards. Each card holds 10.51 GiB of weights (measured) and dequantizes only half of them on every forward pass. All shard boundaries land on whole 32-element MX blocks (checked against the real shapes), and both cards carry 12 attention heads, 2 KV heads, 8 GDN key heads and 24 value heads.
• NCCL_PROTO=Simple and --attention-backend TRITON_ATTN are what the community R9700 TP=2 / TP=4 setups use (Level1Techs guide); a TP=2 hang on dual R9700 is still reported open (vllm#40980), which is why Recipe Z exists. NCCL_P2P_DISABLE=1 is kept for robustness against hipIpc errors (the guide found it unnecessary). vLLM's custom all-reduce is MI300/MI350-only, so Radeon cards talk through RCCL over PCIe.
• No speculation, prefix caching on. This recipe avoids every speculative-decoding issue, so it is the most likely TP=2 recipe to work on the first try, and it is the reference for the tester matrix. Its stock-emulation form (AEON_MXFP6_ONCE=0) is community-tested on 2x R9700: it started at these defaults, passed the arithmetic and tool-call smoke tests and decoded at about 1.7 tok/s (Measured).
• --max-model-len 131072 --max-num-seqs 16. With the MXFP6 BF16 copy (about 13.7 GiB of weights per card), graphs (2.0–2.5 GiB) and activations (2.4–3.6 GiB), an estimated 8.9–10.6 GiB per card is left for cache: BF16 KV costs 32 KiB per token per card, plus about 0.14 GiB of Gated DeltaNet state per running request per card. One 131,072-token request needs about 4.2 GiB (Maximum concurrency about 2.1–2.6x), a 16K-token request about 0.65 GiB (about 13–16 at once); expect GPU KV cache size around 280–335K tokens. The startup line is the real number. For one full 262,144-token sequence, use the stock path: AEON_MXFP6_ONCE=0 AEON_MAX_MODEL_LEN=262144 AEON_MAX_NUM_SEQS=2 (what 262K means). AEON_MXFP6_ONCE=0 gives back about 3.1 GiB per card, at roughly a third to a quarter of the speed (about 1.7 vs an estimated 6 tok/s).
• --language-model-only skips the vision tower (it fails to load on RDNA 4 today).
• --gpu-memory-utilization 0.90: use 0.85–0.88 if a card drives a display.

Recipe Z: 2x R9700 as two independent servers (fallback to test)

Target: 2x AMD Radeon AI PRO R9700 · VRAM: 2 × 32 GB (one full copy per card) · Arch: gfx1201 (RDNA 4) · MX path: emulated, stock (no patch) · Status: Expected (fewest moving parts; each replica's TP=1 eager setup is community-tested as Recipe B, the pair is not yet) · Script: scripts/serve_2x_r9700_replicas.sh (stop with ... replicas.sh stop)

Use this when every TP=2 recipe fails twice (hipIpcGetMemHandle error, crash at graph capture), or right away after a TP=2 hang (a serve script exits with code 3, "PROBABLY HUNG"; RCCL, vllm#40980, ROCm/ROCm#6148): a hang doesn't depend on speculative decoding, so A-MTP and A-plain would hang the same way. It runs one complete copy of the model per card, with no inter-GPU communication, no speculative decoding, no graph capture (--enforce-eager) and no AITER: aeon-mxfp4-0 on port 8000, aeon-mxfp4-1 on port 8001, and a small least-connections proxy (scripts/aeon_lb.py, or scripts/nginx_aeon_replicas.conf) on port 8080. The MXFP6 dequant-once patch doesn't fit next to a full copy of the weights on one 32 GB card, so Z runs stock emulation: it is the slowest option, a last resort.

bash "$AEON_DIR/scripts/serve_2x_r9700_replicas.sh"

Plain docker run for Recipe Z

for i in 0 1; do
docker run -d --name aeon-mxfp4-$i \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:$((8000+i)):8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1201 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_r9700_replicas \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e HIP_VISIBLE_DEVICES=$i -e VLLM_ROCM_USE_AITER=0 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 32768 \
  --max-num-seqs 4 --gpu-memory-utilization 0.88 --kv-cache-dtype auto --max-num-batched-tokens 4096 \
  --attention-backend TRITON_ATTN --enforce-eager --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'
[ "$i" = 0 ] && until docker logs aeon-mxfp4-0 2>&1 | grep -q "Application startup complete"; do
  docker inspect -f '{{.State.Running}}' aeon-mxfp4-0 2>/dev/null | grep -q true || break; sleep 15
done   # GPU 1 reuses the kernel cache GPU 0 built; stops waiting if replica 0 died
done
python3 "$AEON_DIR/scripts/aeon_lb.py" --listen 127.0.0.1:8080 127.0.0.1:8000 127.0.0.1:8001 &

• Cost (measured analog): these are the memory flags of the community-tested Recipe B run (TP=1, eager, 0.88, 4,096 batched tokens; only the context differs): 20.76 GiB of weights and a 5.15 GiB pool per card (64,929 tokens), so about 2.2 concurrent 32K-token or about 4 concurrent 16K-token requests per card. Speed: 0.86 tok/s per stream (measured; stock MXFP6 emulation, TP=1, eager).
• The script starts the second server only after the first is ready, so the one-time kernel build isn't run twice at the same time. Each container gets only its own GPU's device nodes (detect_setup.py --device-flags --index N), so an iGPU never enters a replica. The plain docker run above passes all of /dev/dri with HIP_VISIBLE_DEVICES=$i: on a host with an iGPU, replace --device /dev/dri with the nodes from python3 "$AEON_DIR/scripts/detect_setup.py" --device-flags --index $i and drop HIP_VISIBLE_DEVICES.
• Keep --enforce-eager. Compiled TP=1 on a 32 GB card measured 0.29 GiB of KV cache (the start fails: Measured). Eager vs graphs has not been A/B-measured; TP=1 eager at 0.86 tok/s against TP=2 with graphs at ~1.7 is consistent with eager costing little on the stock path.
• Don't use --data-parallel-size for this: it still sets up inter-GPU process groups.

Agent clients and the prefix cache

At startup vLLM 0.31 may log: "The chat template requires system-first ordering, so inline system messages in /v1/messages requests … are merged into the leading system prompt." (reported by Todd (community tester)). It is a startup check, not an error. It only matters if (1) your client sends role: "system" entries inside messages[] on the Anthropic-compatible /v1/messages endpoint, for example a per-turn reminder, and (2) prefix caching is on (A-MTP, A-plain and the other non-DFlash2 recipes; DFlash2 recipes re-prefill every turn anyway). Then each new reminder is moved into the leading system prompt, so every turn re-prefills the whole conversation, which is slow under MX emulation. (On /v1/chat/completions, the stock template rejects a later system message with an error.)

Two fixes; the default stays unchanged:

• Client side (no server change): send per-turn reminders as a text block inside the user message rather than as a separate system message. The stock template then keeps the prefix stable.
• Server side (opt-in): AEON_CHAT_TEMPLATE=agentic bash "$AEON_DIR/scripts/serve_2x_r9700_mtp.sh" (in a plain docker run: --chat-template /model/scripts/chat_template_agentic.jinja). scripts/chat_template_agentic.jinja renders byte-identically to the model's chat_template.jinja for every conversation without a later system message (8 tested cases: plain, system + user, tool loops, thinking off, reasoning effort, list content). A later system message is rendered in place as its own system turn (the Qwen2.5 / Qwen3 convention) instead of being rejected, so vLLM stops merging, the startup warning goes away and earlier turns keep their cache entries (in a 6-turn test render, 100% of the previous turn's prompt was reused each turn, against 96% falling to 45% by the sixth turn with the stock template). Qwen3.5-generation templates reject later system messages, so this layout is outside the model's chat-template training: answer quality with it has not been measured on AEON yet; A/B it with your client before relying on it.

To check: vLLM's periodic Prefix cache hit rate log line and the vllm:prefix_cache_hits / vllm:prefix_cache_queries counters on /metrics should rise across the turns of one conversation. Reuse comes in whole cache blocks (784 tokens on this model in the community test log), so up to one block plus the new turn is re-prefilled each turn even when it works.

Tester matrix: plain vs MTP vs DFlash2

mkdir -p ~/aeon-test && cd ~/aeon-test && bash "$AEON_DIR/scripts/run_matrix.sh"

It starts A-plain, A-MTP and A one after another (all with 16 sequences), runs the decode sanity test on each (an arm below 5 tok/s single-stream skips its long eval unless AEON_MATRIX_FORCE=1; the A-plain arm, estimated at ~6 tok/s, is gated at 3 so the exact baseline isn't dropped: AEON_MATRIX_PLAIN_MIN_TOK_S), runs the same checks on each, then checks TP=2 against a one-GPU reference, and writes results/matrix_compare.md, results/tpcheck.md and the archive to send back:

The three should score the same on the quality checks within noise: speculative decoding doesn't change the output distribution. It is not bit-reproducible, though: at T=0 the outputs usually diverge after a few dozen tokens, because verify batches change the floating-point reduction order (measured: 1 of 20 GSM8K outputs identical to plain decoding, mean common prefix 16%), while accuracy stayed statistically the same (plain GSM8K 20/20 and tools 10/10, DFlash2 19/20 and 9/10: two discordant items, McNemar p = 0.5). Plain decoding at different concurrency isn't bit-reproducible either. Expect 1–3 hours for all three arms after the first compile; AEON_MATRIX_ARMS="dflash2 mtp" runs a subset. An arm still compiling after the extra wait stops the matrix instead of being killed mid-build.

Recipe A-Max: 2x R9700, BF16-resident MLP (experimental, superseded)

Target: 2x AMD Radeon AI PRO R9700 · VRAM: 2 × 32 GB · Arch: gfx1201 (RDNA 4) · MX path: MXFP4 MLP dequantized once at load; MXFP6 on stock per-pass emulation; MTP in BF16 · Status: Tested (community): 32,768 context, 5.5 tok/s; 65,536 runs out of KV · Script: scripts/serve_2x_r9700_max.sh

vLLM 0.31's VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD=1 stores the MXFP4 MLP as BF16 (output identical; +11.9 GiB per GPU, measured). It keeps the MXFP6 projections on stock emulation, which costs about 6x more per pass (measured) than the MXFP4 emulation this removes, and both BF16 copies together don't fit on 32 GB cards. Expect it to be slower than A-MTP. It stays in the kit as an A/B data point only (32,768 context, 4 sequences, --gpu-memory-utilization 0.92). Community-measured: 22.8 GiB of weights (incl. MTP) and 2.01 GiB of graphs per GPU; 65,536 failed with 1.76 GiB of cache against 2.52 GiB needed, and 49,152 is above the engine's own estimate of about 42,400 tokens, so don't try it; 32,768 ran at 5.5 tok/s single-stream and 9.6 tok/s total at 2 requests (Measured). The community-tested optimized runtime replaced this BF16-resident MLP with packed W4A4 (below); its integration into this kit is in progress.

Community-tested optimized runtime (integration in progress)

Todd (community tester) built a vLLM v0.31.0 runtime on the pinned image that serves this checkpoint unchanged, with its own W4A4/W6A6 quantization, at 49.8 tok/s single-request and 109 tok/s total at 2 requests on 2x R9700 (TP=2, MTP 3, 65,536 context; Measured). Two changes do it:

• MXFP6 dequantized once at load, activation MXFP6 quantize-dequantize kept. This kit's MXFP6 dequant-once patch is the same configuration (AEON_MXFP6_ONCE=1, on by default in A, A-MTP and A-plain), so you already have this half.
• The MXFP4 MLP kept packed and run as W4A4 on gfx1201 through AITER's Triton scaled GEMM (BF16 WMMA inside; activations stay MXFP4, not W4A8) with 16-wide RDNA 4 tile configs, enabled for that one op only. Stock vLLM rejects AITER on gfx1201 and falls back to emulation. The tile table was adapted from GGZ14/vllm-mxfp4 (commit 5416b1a), which has no license file, so that part ships here only once the licensing is settled (or as our own tuning).

Its integration into this kit is in progress. Until it lands, nothing in scripts/ turns it on, and there are no flags for it to set. In particular, don't set AEON_AITER=1 to get it: that runtime keeps VLLM_ROCM_USE_AITER=0 and enables only the W4A4 op, while VLLM_ROCM_USE_AITER=1 turns on AITER kernels that fail on RDNA 4.

AITER file locations (reported by Todd; verify in your image): in the pinned v0.31.0 image, AITER's MXFP4 W4A4 Triton GEMM is aiter.ops.triton.gemm.basic.gemm_afp4wfp4, and per-architecture tile tables live under configs/<arch>/triton/gemm/gemm_afp4wfp4/ (gfx1201: configs/gfx1201/triton/gemm/gemm_afp4wfp4/DEFAULT.json). Older AITER used configs/gemm/<arch>-*.json. vLLM 0.31's own MXFP4 AITER kernel (mxfp4/aiter.py, used only on MI350 / MI355) imports the older module path aiter.ops.triton.gemm_afp4wfp4. The launchers record which of these files your image has in image_check_<recipe>.txt (AITERPATH lines).

Speed experiment: radiance (experimental, unlicensed, untested on this checkpoint)

radiance is a community overlay for vLLM with native RDNA 4 kernels: MXFP4 weights with FP8 activations (W4A8) and E6 (MXFP6) kernels. Users report 130–210 tok/s single-stream with DFlash2 on 2x R9700 with other checkpoints. It is not part of this kit, and we have not run it with this checkpoint.

• It can't serve this checkpoint as shipped. Its MXFP6 kernels are wired only to its own paroquant_mxfp6 format. Serving our Quark MXFP6 layers needs a new kernel class in vLLM's MXFP6 kernel list. Our weights would fold exactly: every MXFP6 row has an E8M0 scale spread of at most 6, which its fold requires (measured on all 1,032,192 rows). Its Quark MXFP4 patch targets vLLM 0.29, which can't load this checkpoint without AEON_STRIP_ALGO_CONFIG=1.
• It changes the numerics. Activations become per-token FP8 instead of the checkpoint's MXFP4 / MXFP6, so the output is not identical to Recipes A, A-MTP and A-plain (radiance's own MXFP4 perplexity is slightly better with W4A8, but it is a deviation). Its defaults also include lossy options that must be turned off for a fidelity comparison: quantized all-reduce, FP8 KV cache and a quantized output head.
• License. The original repository has no license file. Private experiments only; don't redistribute it or code copied from it.
• If you try it, compare it against A-plain with tests/run_radeon_eval.py --compare, and send us both reports.

TensorFold

TensorFold (Apache-2.0) drafts with DFlash2 and is bit-exact against its own serial decoding. Its ROCm support is a community fork (millaguie/TensorFold, branch rocm-r9700).

• It can't load this mixed checkpoint, so it is not a path for this model today. The ROCm fork serves Qwen3.8-27B MLX 4-bit checkpoints, GGUF, or a Quark export where every projection is MXFP4, one GPU per instance (no TP=2). Our checkpoint is detected as Quark MXFP4 and then fails at the first MXFP6 or BF16 projection; it would need an MXFP6 path, mixed projection groups and BF16 linears on ROCm. Pairing the drafter with a stock Qwen3.8-27B checkpoint would serve a different model. On 2x R9700, use vLLM Recipe A.
• The drafter is shipped for TensorFold so it can be used as soon as TensorFold supports this mixed Quark export: tensorfold serve <AEON target> --drafter "$AEON_DIR/dflash2". Its config and tensors load in TensorFold's DFlash2 loader (checked against the source and the fork's reader). TensorFold's CUDA/ROCm engine ignores --drafter-bits and re-packs the drafter's linears to affine 4-bit (group 64) at load, as it does for z-lab's drafter, which can lower acceptance but not output fidelity. See the drafter card.

Recipe K: 4x Radeon AI PRO R9700 with the AEON DFlash2 drafter

Target: 4x AMD Radeon AI PRO R9700 / R9700S / R9600D · VRAM: 4 × 32 GB · Arch: gfx1201 (RDNA 4) · MX path: emulated, MXFP6 dequantized once and MXFP4 MLP kept in BF16 after load (bit-identical); drafter in BF16 · Status: Untested · Script: scripts/serve_4x_r9700.sh

bash "$AEON_DIR/scripts/serve_4x_r9700.sh"

Plain docker run for Recipe K

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/dflash2":/draft:ro \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1201 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-4x_r9700 \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 -e VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD=1 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 -e VLLM_USE_V2_MODEL_RUNNER=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 4 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 8192 \
  --attention-backend TRITON_ATTN --no-enable-prefix-caching --language-model-only \
  --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":9,"num_speculative_tokens_per_batch_size":[[1,2,9],[3,8,7],[9,12,6],[13,16,4]],"attention_backend":"TRITON_ATTN","draft_sample_method":"probabilistic","rejection_sample_method":"standard"}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate) per GPU: weights about 5.4 + MXFP6 BF16 net 1.6 + BF16 MLP 6.0 + drafter 0.9 = 13.8 GiB, graphs about 1.3–2.2 and activations about 1.3–2.3 GiB, about 10.4–12.3 GiB of pool (about 20 KiB per token per GPU: 16 KiB, padded to 20 for the drafter's layers; Gated DeltaNet state comes out of the same pool). One 131,072-token DFlash2 request needs about 2.9 GiB: about 3.5–4 at once (about 460–550K tokens in the log). K-MTP at 262,144 needs about 4.5 GiB per request (about 2.6–3x).
• Speed (estimate): about 25–30 tok/s plain and 60–90 with DFlash2 single-stream (±50%; bandwidth model, read as the top of the range: it was about 1.7x optimistic for the patched path on 2x R9700).
• Head counts divide: 1 KV head and 6 attention heads per GPU for the target, 2 KV heads for the drafter. TP=4 with NCCL_PROTO=Simple is reported working on 4x R9700 with other checkpoints.
• Variants: K-MTP for long multi-turn agent loops (prefix caching on): AEON_DFLASH= AEON_MTP=3 AEON_MAX_MODEL_LEN=262144 AEON_MAX_NUM_SEQS=8. K-plain: AEON_DFLASH=.
• Fallbacks: two Recipe A servers, one per pair of cards (AEON_DEVICES, AEON_NAME, AEON_PORT 8000 / 8001, AEON_REPLACE_OTHERS=0 for the second), then Z with AEON_REPLICAS=4.

Recipe B: 1x Radeon AI PRO R9700

Target: 1x AMD Radeon AI PRO R9700 / R9700S / R9600D · VRAM: 32 GB · Arch: gfx1201 (RDNA 4) · MX path: emulated, stock, eager · Status: Tested (community): 16,384 context, 0.86 tok/s · Script: scripts/serve_1x_r9700.sh

bash "$AEON_DIR/scripts/serve_1x_r9700.sh"

Plain docker run for Recipe B

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1201 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-1x_r9700 \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e VLLM_ROCM_USE_AITER=0 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 16384 \
  --max-num-seqs 2 --gpu-memory-utilization 0.88 --kv-cache-dtype auto --max-num-batched-tokens 4096 \
  --attention-backend TRITON_ATTN --enforce-eager --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Measured by Todd (community tester) (Measured): these defaults (eager, 16,384 context, 2 sequences, 0.88) start with 20.76 GiB of weights and a 5.15 GiB KV pool = 64,929 tokens (about 4 concurrent 16K-token requests), and decode at 0.86 tok/s (128 tokens, TTFT 2.4 s, first request). A baseline, not a serving setup: for real use, two or more cards.
• The first shipped defaults failed (graphs, 32,768 context, 4 sequences, 0.90): graph capture took 2.50 GiB and left 0.29 GiB of KV against 2.3 GiB needed. Never run one 32 GB card with graphs.
• The MXFP6 dequant-once patch doesn't fit: 20.76 + 6.3 GiB = 27.1, plus about 2.2 GiB of eager overhead, is more than the 28.1 GiB usable at 0.88. So this is stock emulation.
• Longer context (recomputed from the measured pool, estimates): AEON_MAX_MODEL_LEN=32768 needs about 2.3 GiB per request (about 2.2 at once); AEON_MAX_MODEL_LEN=65536 AEON_MAX_NUM_SEQS=1 needs about 4.3 GiB (about 1.2x, tight).
• MTP: AEON_MTP=3 (untested): the MTP weights take 0.79 GiB, leaving about 4.3 GiB, and a 16K MTP request needs about 1.8 GiB (about 2.3 at once).
• Smallest: AEON_MAX_MODEL_LEN=8192 AEON_MAX_NUM_SEQS=1.
• Dequant-at-load and DFlash2 don't fit on one 32 GB card.

Recipe C: 2x 16 GB RDNA 4 (RX 9070 XT, RX 9070, RX 9060 XT)

Target: 2x AMD Radeon RX 9070 XT or RX 9070 (gfx1201), or 2x RX 9060 XT 16 GB (gfx1200, about half the bandwidth) · VRAM: 2 × 16 GB · Arch: gfx1201 / gfx1200 (RDNA 4) · MX path: emulated, stock, eager · Status: Untested (expected to work; tight memory) · Script: scripts/serve_2x_rx9070xt.sh

bash "$AEON_DIR/scripts/serve_2x_rx9070xt.sh"

Plain docker run for Recipe C

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1201 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_rx9070xt \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 2 --max-model-len 16384 \
  --max-num-seqs 2 --gpu-memory-utilization 0.92 --kv-cache-dtype auto --max-num-batched-tokens 2048 \
  --attention-backend TRITON_ATTN --enforce-eager --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Same code path as Recipe A-plain with half the memory: 10.51 GiB of weights per card (measured on R9700 at TP=2) and, eager, an estimated 2.2–3.3 GiB of pool per card (about 3–5 concurrent 16K-token requests); the MXFP6 dequant-once patch doesn't fit. Graphs alone would take 2.0–2.5 GiB per card (measured on R9700) and leave no KV, so the recipe runs eager. Eager vs graphs has not been A/B-measured; on R9700, TP=1 eager at 0.86 tok/s against TP=2 with graphs at about 1.7 is consistent with eager costing little on the stock path. About 2 tok/s single-stream on RX 9070 XT, about 1 on RX 9060 XT (estimates). On RX 9060 XT cards use PYTORCH_ROCM_ARCH=gfx1200 in a plain docker run (the script detects it). If a card drives a display, use --gpu-memory-utilization 0.88. MTP (AEON_MTP=2) only if the GPU KV cache size line shows room. FP8 KV (AEON_KV_DTYPE=fp8) doubles the context but changes outputs (uncalibrated scales); it is opt-in. There is no single-card fallback: the model doesn't fit on 16 GB.

Recipe D: 4x 16 GB cards (RDNA 4 or RDNA 3)

Target: 4x AMD Radeon RX 9070 XT / RX 9070 (gfx1201) or RX 9060 XT 16 GB (gfx1200); or 4x RX 7900 GRE (gfx1100), RX 7800 XT / RX 7700 / Radeon PRO W7700 (gfx1101) · VRAM: 4 × 16 GB · Arch: gfx1201 / gfx1200 / gfx1100 / gfx1101 · MX path: emulated, MXFP6 dequantized once at load · Status: Untested · Script: scripts/serve_4x_rx9070xt.sh

bash "$AEON_DIR/scripts/serve_4x_rx9070xt.sh"

Plain docker run for Recipe D

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1201 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-4x_rx9070xt \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 4 --max-model-len 65536 \
  --max-num-seqs 8 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 4096 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate) per GPU: weights about 5.4 + MXFP6 BF16 net 1.6 GiB, graphs about 1.2–2.0, activations about 1.1–1.8 GiB at 4,096 batched tokens, about 3.6–5.1 GiB of pool (16 KiB per token per GPU): a 64K-token request needs about 1.1 GiB, about 3–4.5 at once. Longer context: AEON_MAX_MODEL_LEN=131072 AEON_MAX_NUM_SEQS=4, only if the log's pool is at least about 2.2 GiB.
• Speed (estimate): about 18 tok/s single-stream on RX 9070 XT, about 40 with MTP (AEON_MTP=3), about 50 with DFlash2 (AEON_DFLASH="$AEON_DIR/dflash2" AEON_MAX_MODEL_LEN=16384 AEON_MAX_NUM_SEQS=2, prefix caching off; check Maximum concurrency is at least 1).
• TP=4 shards check out (6 attention heads and 1 KV head per card; every MX shard on a whole block), and TP=4 with NCCL_PROTO=Simple is reported working on 4x R9700. All-reduce over consumer PCIe likely limits speed. If startup hangs at graph capture, AEON_EAGER=1.
• 4x 12 GB (RX 9070 GRE, RX 7700 XT): AEON_MXFP6_ONCE=0 AEON_EAGER=1 AEON_MAX_BATCHED=4096 AEON_MAX_MODEL_LEN=32768 AEON_MAX_NUM_SEQS=4 AEON_GPU_UTIL=0.92 (untested; eager pool about 4.0–4.8 GiB per card).
• Don't use 8 cards with TP=8: 4 KV heads can't be split 8 ways.

Recipe E: 2x Radeon RX 7900 XTX

Target: 2x AMD Radeon RX 7900 XTX · VRAM: 2 × 24 GB · Arch: gfx1100 (RDNA 3) · MX path: emulated, MXFP6 dequantized once at load · Status: Untested · Script: scripts/serve_2x_rx7900xtx.sh

bash "$AEON_DIR/scripts/serve_2x_rx7900xtx.sh"

Plain docker run for Recipe E

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1100 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_rx7900xtx \
  -e PYTORCH_ROCM_ARCH=gfx1100 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 2 --max-model-len 32768 \
  --max-num-seqs 4 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 4096 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate) per GPU: weights 10.51 (measured on R9700) + MXFP6 BF16 net 3.1 GiB, graphs 2.0–2.5, activations 2.1–2.8 GiB, about 2.6–3.8 GiB of pool (RDNA 3 has no FP8, so KV stays BF16 at 32 KiB per token per card): a 32K-token request needs about 1.15 GiB, about 2.3–3.3 at once. With AEON_MXFP6_KEEP_PACKED=1 the pool shrinks by about 2 GiB: add AEON_EAGER=1 then.
• Speed (estimate): about 15 tok/s single-stream, about 35 with MTP (AEON_MTP=3).
• Variants: AEON_MAX_MODEL_LEN=65536 AEON_MAX_NUM_SEQS=2 (a 64K request needs about 2.15 GiB: thin; check Maximum concurrency is at least 1, else add AEON_EAGER=1). Long context on stock emulation (about 5x slower): AEON_MXFP6_ONCE=0 AEON_MAX_MODEL_LEN=131072 AEON_MAX_NUM_SEQS=2 (pool about 5.8–7.0 GiB against 4.2 needed). DFlash2 and dequant-at-load don't fit next to the patch.
• 2x RX 7900 XT (20 GB): use Recipe M.

Recipe M: 2x Radeon RX 7900 XT

Target: 2x AMD Radeon RX 7900 XT · VRAM: 2 × 20 GB · Arch: gfx1100 (RDNA 3) · MX path: emulated, stock, eager · Status: Untested · Script: scripts/serve_2x_rx7900xt.sh

bash "$AEON_DIR/scripts/serve_2x_rx7900xt.sh"

Plain docker run for Recipe M

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1100 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_rx7900xt \
  -e PYTORCH_ROCM_ARCH=gfx1100 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 2 --max-model-len 32768 \
  --max-num-seqs 4 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 4096 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate) per GPU: about 18 GiB usable, 10.51 GiB of weights (measured on R9700), eager overhead about 1.1–2.2 GiB, about 5.3–6.4 GiB of pool: a 32K-token request needs about 1.15 GiB, about 4.6–5.5 at once. Eager frees the 2.0–2.5 GiB of graphs (measured on R9700), likely at little speed cost on the stock path (not A/B-measured); with AEON_EAGER=0 the pool is about 2.2–3.4 GiB.
• Speed (estimate): about 2.3 tok/s single-stream (stock emulation); MTP AEON_MTP=3.
• Opt-in MXFP6 dequant-once (about 12 tok/s, estimate; tight, untested, no display on the cards): AEON_MXFP6_ONCE=1 AEON_GPU_UTIL=0.93 AEON_MAX_MODEL_LEN=16384 AEON_MAX_NUM_SEQS=2 AEON_MAX_BATCHED=2048 (eager; about 3.0–4.0 GiB of pool per GPU; with graphs it doesn't fit).

Recipe L: 2x 16 GB RDNA 3 (RX 7900 GRE, RX 7800 XT, RX 7700, PRO W7700)

Target: 2x AMD Radeon RX 7900 GRE (gfx1100), or 2x RX 7800 XT / RX 7700 / Radeon PRO W7700 (gfx1101) · VRAM: 2 × 16 GB · Arch: gfx1100 / gfx1101 (RDNA 3) · MX path: emulated, stock, eager · Status: Untested (tight memory) · Script: scripts/serve_2x_16gb_rdna3.sh

bash "$AEON_DIR/scripts/serve_2x_16gb_rdna3.sh"

Plain docker run for Recipe L

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1100 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_16gb_rdna3 \
  -e PYTORCH_ROCM_ARCH=gfx1100 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 2 --max-model-len 16384 \
  --max-num-seqs 2 --gpu-memory-utilization 0.92 --kv-cache-dtype auto --max-num-batched-tokens 2048 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Like Recipe C on RDNA 3: eager, about 2.2–3.3 GiB of pool per card (estimate; a 16K-token request takes about 0.5 GiB of KV plus 0.15 GiB of state per card), stock emulation, about 1.7 tok/s single-stream (estimate). On gfx1101 cards use PYTORCH_ROCM_ARCH=gfx1101 in a plain docker run (the script detects it). Use 0.88 for --gpu-memory-utilization if a card drives a display. MTP (AEON_MTP=2) only if the GPU KV cache size line shows at least 1.5 GiB. Smallest: AEON_MAX_MODEL_LEN=8192 AEON_MAX_NUM_SEQS=1. No single-card fallback.

Recipe F: Radeon PRO W7900 (48 GB)

Target: 1x AMD Radeon PRO W7900, W7900 Dual Slot or W7800 48 GB · VRAM: 48 GB · Arch: gfx1100 (RDNA 3) · MX path: emulated, MXFP6 dequantized once at load · Status: Untested · Script: scripts/serve_w7900.sh

bash "$AEON_DIR/scripts/serve_w7900.sh"

Plain docker run for Recipe F

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1100 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-w7900 \
  -e PYTORCH_ROCM_ARCH=gfx1100 -e VLLM_ROCM_USE_AITER=0 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 65536 \
  --max-num-seqs 4 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 8192 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate): weights 20.76 (measured on R9700) + MXFP6 BF16 net 6.3 = 27.0 GiB, graphs 2.0–2.5, TP=1 compiled activations about 4.6–6.8 GiB (extrapolated from the measured R9700 TP=1 run), about 6.9–9.5 GiB of pool (64 KiB per token): a 64K-token request needs about 4.3 GiB, about 1.6–2.2 at once. With AEON_MXFP6_KEEP_PACKED=1 the pool shrinks by about 4 GiB: use AEON_MAX_MODEL_LEN=32768 AEON_MAX_BATCHED=4096 then. Dequant-at-load doesn't fit (20.76 + 23.8 GiB > 43.2 usable).
• Speed (estimate): about 8 tok/s single-stream, about 18 with MTP (AEON_MTP=3), about 25 with DFlash2 (AEON_DFLASH="$AEON_DIR/dflash2" AEON_EAGER=1 AEON_MAX_MODEL_LEN=32768 AEON_MAX_NUM_SEQS=2 AEON_MAX_BATCHED=4096, prefix caching off; with graphs it doesn't fit).
• Variants: AEON_MAX_MODEL_LEN=131072 AEON_MAX_NUM_SEQS=2, only if the default start logs a pool of at least about 8.4 GiB. The full 262,144-token context on stock emulation (about 5x slower): AEON_MXFP6_ONCE=0 AEON_EAGER=1 AEON_MAX_MODEL_LEN=262144 AEON_MAX_NUM_SEQS=1 (eager pool about 19.6–20.4 GiB against 16.3 needed; with graphs it doesn't fit). Vision: AEON_VISION=1.
• Two or four of these cards: Recipe Q.

Recipe Q: 2x Radeon PRO W7900 with the AEON DFlash2 drafter

Target: 2x AMD Radeon PRO W7900, W7900 Dual Slot or W7800 48 GB (variants for 4x W7900 and 2x W7800 32 GB) · VRAM: 2 × 48 GB · Arch: gfx1100 (RDNA 3) · MX path: emulated, MXFP6 dequantized once and MXFP4 MLP kept in BF16 after load (bit-identical); drafter in BF16 · Status: Untested · Script: scripts/serve_2x_w7900.sh

bash "$AEON_DIR/scripts/serve_2x_w7900.sh"

Plain docker run for Recipe Q

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/dflash2":/draft:ro \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1100 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-2x_w7900 \
  -e PYTORCH_ROCM_ARCH=gfx1100 -e NCCL_PROTO=Simple -e NCCL_P2P_DISABLE=1 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ROCM_USE_AITER=0 -e VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD=1 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 -e VLLM_USE_V2_MODEL_RUNNER=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 2 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 8192 \
  --attention-backend TRITON_ATTN --no-enable-prefix-caching --language-model-only \
  --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":9,"num_speculative_tokens_per_batch_size":[[1,2,9],[3,8,7],[9,12,6],[13,16,4]],"attention_backend":"TRITON_ATTN","draft_sample_method":"probabilistic","rejection_sample_method":"standard"}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate) per GPU: 10.51 + 3.1 (MXFP6 BF16 net) + 11.9 (BF16 MLP) + 1.8 (drafter) = 27.3 GiB, graphs about 1.6–2.6 and activations about 2.4–3.6 GiB, about 9.6–11.8 GiB of pool: a 131,072-token DFlash2 request needs about 5.8 GiB (about 1.6–2.0 at once).
• Speed (estimate): about 24 tok/s plain and about 70 with DFlash2 single-stream.
• Variants: Q-MTP for long multi-turn agent loops (prefix caching on): AEON_DFLASH= AEON_MTP=3 AEON_MAX_MODEL_LEN=262144 AEON_MAX_NUM_SEQS=8 (a 262K MTP request needs about 8.9 GiB of an 11.6–13.5 GiB pool; check Maximum concurrency is at least 1). 2x W7800 32 GB: AEON_DEQUANT_AT_LOAD=0 AEON_DFLASH= (like A-plain; about 8.9–10.6 GiB of pool per card). 4x W7900: AEON_TP=4.
• Fallback: one Recipe F server per card (AEON_GPU_IDS=0 / 1, AEON_PORT 8000 / 8001, AEON_NAME, AEON_REPLACE_OTHERS=0 for the second).

Recipe G: Radeon PRO W7800 (32 GB)

Target: 1x AMD Radeon PRO W7800 32 GB · VRAM: 32 GB · Arch: gfx1100 (RDNA 3) · MX path: emulated, stock, eager · Status: Untested (slow; uses Recipe B's community-tested settings) · Script: scripts/serve_w7800_32gb.sh

bash "$AEON_DIR/scripts/serve_w7800_32gb.sh"

Plain docker run for Recipe G

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1100 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-w7800_32gb \
  -e PYTORCH_ROCM_ARCH=gfx1100 -e VLLM_ROCM_USE_AITER=0 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 16384 \
  --max-num-seqs 2 --gpu-memory-utilization 0.88 --kv-cache-dtype auto --max-num-batched-tokens 4096 \
  --attention-backend TRITON_ATTN --enforce-eager --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Like Recipe B on RDNA 3, with B's community-tested settings (eager, 16,384 context, 2 sequences, 0.88): an estimated 5.1 GiB of pool for BF16 KV and Gated DeltaNet state (about 4 concurrent 16K-token requests; 32K with AEON_MAX_MODEL_LEN=32768, about 2.2 at once). B's first defaults (graphs, 32,768, 4 sequences, 0.90) failed on the R9700 with 0.29 GiB of KV left, and this card has the same memory. The MXFP6 dequant-once patch doesn't fit, so about 0.8 tok/s single-stream (0.86 measured on R9700, scaled by bandwidth; estimate). MTP: AEON_MTP=3. Text-only. For real use, two cards (Recipe Q's 2x W7800 32 GB variant).

Recipe P: Radeon PRO V710

Target: 1x AMD Radeon PRO V710 (the full GPU, not a fractional partition; usually a cloud VM) · VRAM: 28 GB · Arch: gfx1101 (RDNA 3) · MX path: emulated, stock, eager · Status: Untested (slow) · Script: scripts/serve_v710.sh

bash "$AEON_DIR/scripts/serve_v710.sh"

Plain docker run for Recipe P

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1101 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-v710 \
  -e PYTORCH_ROCM_ARCH=gfx1101 -e VLLM_ROCM_USE_AITER=0 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 16384 \
  --max-num-seqs 2 --gpu-memory-utilization 0.92 --kv-cache-dtype auto --max-num-batched-tokens 2048 \
  --attention-backend TRITON_ATTN --enforce-eager --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

About 25.7 GiB usable, 20.76 GiB of weights (measured on R9700), eager overhead about 1.4–2.0 GiB, about 3.0–3.5 GiB of pool (estimate; a 16K-token request takes about 1.0 GiB of KV plus 0.29 GiB of state, about 2.3–2.7 at once). Eager is required here: graphs (2.0–2.5 GiB, measured on R9700) plus the compiled activation peak would leave no KV. About 0.6 tok/s single-stream (0.86 measured on R9700, scaled by bandwidth; estimate). No room for speculation. TP=1 needs no RCCL, so the preflight's virtual-machine warning doesn't apply here. Smallest: AEON_MAX_MODEL_LEN=8192 AEON_MAX_NUM_SEQS=1.

Recipe H: Strix Halo (Ryzen AI Max+ 395), 64 GB

Target: AMD Ryzen AI Max+ 395 / Max 390 / Max 385 APU (Radeon 8060S / 8050S) with 64 GB · Memory: 64 GB unified (at least 45 GiB GPU-addressable) · Arch: gfx1151 (RDNA 3.5) · MX path: emulated, MXFP6 dequantized once at load, eager · Status: Untested · Script: scripts/serve_strix_halo.sh

Before you start, the GPU must be able to allocate at least about 45 GiB. vLLM sizes its memory from what HIP reports, so raise the TTM/GTT limit the way AMD documents it for Strix Halo (amd-ttm, or the ttm pages_limit module parameter; amdgpu.gttsize is deprecated), or raise the UMA frame buffer in the BIOS. scripts/detect_setup.py prints both values. Don't set HSA_OVERRIDE_GFX_VERSION (older Strix Halo guides do): gfx1151 is a native target of the image, and the launcher refuses the override.

bash "$AEON_DIR/scripts/serve_strix_halo.sh"

Plain docker run for Recipe H

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1151 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-strix_halo \
  -e PYTORCH_ROCM_ARCH=gfx1151 -e VLLM_ROCM_USE_AITER=0 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 32768 \
  --max-num-seqs 2 --gpu-memory-utilization 0.85 --kv-cache-dtype auto --max-num-batched-tokens 4096 \
  --attention-backend TRITON_ATTN --enforce-eager --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate): of about 48 GiB addressable, about 40.8 usable: weights 20.76 (measured on R9700) + MXFP6 BF16 net 6.3 = 27.0 GiB, eager overhead about 1.8–2.2 GiB, about 11.6–12.0 GiB of pool (a 32K-token request needs about 2.3 GiB). BF16 KV (no FP8 on gfx1151).
• Eager by default: at about 256 GB/s a forward pass takes at least about 0.4 s, so graph-launch savings are negligible, while graphs plus the TP=1 compiled activation peak (measured on R9700) would cost about 4–5 GiB of pool. AEON_EAGER=0 to compare.
• Speed (estimate): about 2.3 tok/s single-stream (about 0.4 without the patch), about 5 with MTP (AEON_MTP=3).
• Precision caveat: AMD lists only FP16 as officially validated on Ryzen APUs. This recipe runs BF16 (the reference numerics; fp16 also breaks the DFlash2 drafter): an untested risk.
• Smallest: AEON_MAX_MODEL_LEN=16384 AEON_MAX_NUM_SEQS=1. If vLLM's free-memory check fails on unified memory, AEON_EXTRA_ARGS="--kv-cache-memory-bytes <bytes>" is an untested alternative.

Recipe H-128: Strix Halo (Ryzen AI Max+ 395), 128 GB

Target: AMD Ryzen AI Max+ 395 / Max 390 / Max 385 APU with 128 GB · Memory: 128 GB unified (at least 90 GiB GPU-addressable) · Arch: gfx1151 (RDNA 3.5) · MX path: emulated, MXFP6 dequantized once and MXFP4 MLP kept in BF16 after load (bit-identical); MTP head in BF16 · Status: Untested · Script: scripts/serve_strix_halo_128gb.sh

Same preparation as Recipe H, with at least 90 GiB GPU-addressable.

bash "$AEON_DIR/scripts/serve_strix_halo_128gb.sh"

Plain docker run for Recipe H-128

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx1151 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-strix_halo_128gb \
  -e PYTORCH_ROCM_ARCH=gfx1151 -e VLLM_ROCM_USE_AITER=0 -e VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD=1 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 131072 \
  --max-num-seqs 4 --gpu-memory-utilization 0.80 --kv-cache-dtype auto --max-num-batched-tokens 8192 \
  --attention-backend TRITON_ATTN --language-model-only \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate): of about 96 GiB addressable, about 76.8 usable: weights 20.76 + 6.3 (MXFP6 BF16 net) + 23.8 (BF16 MLP) + MTP 0.8 = 51.6 GiB, graphs about 1.6–2.5 and activations about 4.6–6.8 GiB, about 16–19 GiB of pool: a 131,072-token MTP request needs about 9.3 GiB, about 1.7–2.0 at once.
• Speed (estimate): about 4.5 tok/s plain and about 10 with MTP (the default) single-stream.
• DFlash2 instead of MTP: AEON_MTP=0 AEON_DFLASH="$AEON_DIR/dflash2" AEON_MAX_MODEL_LEN=65536 (prefix caching off). Plain: AEON_MTP=0. Most robust: AEON_EAGER=1 AEON_MAX_MODEL_LEN=32768 AEON_MAX_NUM_SEQS=1.

Recipe I: Instinct MI300X / MI325X

Target: AMD Instinct MI300X, MI308X or MI325X · VRAM: 192 / 256 GB · Arch: gfx942 (CDNA 3) · MX path: emulated (no MX hardware), MXFP6 dequantized once and MXFP4 MLP kept in BF16 after load (bit-identical) · Status: Untested · Script: scripts/serve_mi300x.sh

bash "$AEON_DIR/scripts/serve_mi300x.sh"

Plain docker run for Recipe I

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx942 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-mi300x \
  -e PYTORCH_ROCM_ARCH=gfx942 -e VLLM_ROCM_USE_AITER=0 -e VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD=1 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 262144 \
  --max-num-seqs 32 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 16384 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate), MI300X: about 172 GiB usable, weights 50.8 GiB (20.76 + 6.3 MXFP6 BF16 net + 23.8 BF16 MLP), about 110–114 GiB of pool (about 1.8M BF16 KV tokens): a 262,144-token request needs about 16.3 GiB, about 7 at once (activation figures at 16,384 batched tokens are extrapolated). Memory is not a constraint, so use one GPU per copy and scale out with data parallelism (AEON_DP=8) instead of tensor parallelism.
• Faster: AEON_MTP=3 (best for agentic multi-turn: prefix caching stays on). For short-context throughput, AEON_DFLASH="$AEON_DIR/dflash2" (prefix caching off; with AEON_DP>1 vLLM drops the batch-size lattice, so add AEON_DFLASH_K=6).
• Images: AEON_VISION=1 (4 images per prompt).
• An FP8 or BF16 build is the better everyday choice on MI300, since memory isn't the limit there.

Recipe O: Instinct MI300A

Target: AMD Instinct MI300A (APU) · Memory: 128 GB unified HBM3, shared with the host OS · Arch: gfx942 (CDNA 3) · MX path: emulated, MXFP6 dequantized once and MXFP4 MLP kept in BF16 after load (bit-identical) · Status: Untested · Script: scripts/serve_mi300a.sh

bash "$AEON_DIR/scripts/serve_mi300a.sh"

Plain docker run for Recipe O

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx942 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-mi300a \
  -e PYTORCH_ROCM_ARCH=gfx942 -e VLLM_ROCM_USE_AITER=0 -e VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD=1 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.70 --kv-cache-dtype auto --max-num-batched-tokens 16384 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate): --gpu-memory-utilization 0.70 leaves room for the OS: about 89 GiB usable, weights 50.8 GiB, about 27–32 GiB of pool: a 131,072-token request needs about 8.3 GiB, about 3.3–3.8 at once. Size the utilization from the GiB the launcher prints for the GPU; if it reports much less than 128 GiB, the GPU-allocatable limit has to be raised per AMD's MI300A guidance (ask your administrator).
• Faster: AEON_MTP=3, or DFlash2 (AEON_DFLASH="$AEON_DIR/dflash2", prefix caching off). Several APUs: AEON_DP=<number of APUs>.

Recipe J: Instinct MI350X / MI355X

Target: AMD Instinct MI350X or MI355X · VRAM: 288 GB · Arch: gfx950 (CDNA 4) · MX path: default exact: MXFP4 emulated with the MLP kept in BF16 after load, MXFP6 dequantized once (bit-identical to every other recipe); native MXFP4 (AITER W4A4) opt-in · Status: Untested · Script: scripts/serve_mi355x.sh

bash "$AEON_DIR/scripts/serve_mi355x.sh"

Plain docker run for Recipe J

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx950 \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-mi355x \
  -e PYTORCH_ROCM_ARCH=gfx950 -e VLLM_ROCM_USE_AITER=0 -e VLLM_DISABLED_KERNELS=AiterMxfp4LinearKernel -e VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD=1 \
  -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 262144 \
  --max-num-seqs 32 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 16384 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Why the exact path is the default. Native MXFP4 runs the MLP on AITER's W4A4 GEMM: the same MX grid and E8M0 scale rule, but a different kernel and summation order, so it is not bit-identical to the other recipes. This kit puts fidelity first, so native is a labelled opt-in: AEON_MXFP4_EMULATE=0 AEON_DEQUANT_AT_LOAD=0 (faster at high concurrency; log line Using AiterMxfp4LinearKernel for MXFP4 GEMM). vLLM 0.31 imports that GEMM as aiter.ops.triton.gemm_afp4wfp4, but the community 2x R9700 test found the pinned image's AITER keeps it at aiter.ops.triton.gemm.basic.gemm_afp4wfp4, so the native path may fail with a ModuleNotFoundError; the launcher's image_check_<recipe>.txt shows which file exists. If it does, stay on the default.
• Check the log for Using EmulationMxfp4LinearKernel for MXFP4 GEMM (default) and Using EmulationMxfp6LinearKernel for MXFP6 GEMM.
• Memory (estimate): about 259 GiB usable, weights 50.8 GiB (27.0 with native MXFP4), about 197–224 GiB of pool.
• Faster: DFlash2 (AEON_DFLASH="$AEON_DIR/dflash2") or MTP (AEON_MTP=3). Scaling out: AEON_DP=N (with DFlash2, add AEON_DFLASH_K=6).

Recipe N: Instinct MI210 / MI250 / MI250X

Target: AMD Instinct MI210 (64 GB), MI250 / MI250X (two 64 GB GCDs, each shown as its own GPU) · VRAM: 64 GB per GCD · Arch: gfx90a (CDNA 2) · MX path: emulated, MXFP6 dequantized once at load · Status: Untested · Script: scripts/serve_mi250.sh

bash "$AEON_DIR/scripts/serve_mi250.sh"

Plain docker run for Recipe N

docker run -d --name aeon-mxfp4 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add "$RENDER_GID" \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v "$AEON_DIR":/model:ro -v "$AEON_CACHE":/root/.cache \
  -v "$AEON_DIR/scripts/patches/vllm-0.31.0/mxfp6_emulation.py":/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mxfp6/emulation.py:ro \
  -e TRITON_CACHE_DIR=/root/.cache/triton -e TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions/gfx90a \
  -e VLLM_CACHE_ROOT=/root/.cache/vllm/recipe-mi250 \
  -e PYTORCH_ROCM_ARCH=gfx90a -e VLLM_ROCM_USE_AITER=0 -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  "$AEON_IMAGE" \
  /model --served-model-name aeon --dtype bfloat16 --tensor-parallel-size 1 --max-model-len 131072 \
  --max-num-seqs 8 --gpu-memory-utilization 0.90 --kv-cache-dtype auto --max-num-batched-tokens 8192 \
  --attention-backend TRITON_ATTN --language-model-only \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

• Memory (estimate) per GCD: about 57.6 GiB usable, weights 20.76 + MXFP6 BF16 net 6.3 = 27.0 GiB, graphs about 2.0–2.5 and activations about 4.6–6.8 GiB, about 21–24 GiB of pool: a 131,072-token request needs about 8.3 GiB, about 2.6–2.9 at once. MXFP4 dequant-at-load doesn't fit next to it on one GCD.
• Speed (estimate): about 14 tok/s per GCD single-stream (likely launch-bound), about 33 with MTP (AEON_MTP=3).
• Scale out: one replica per GCD with AEON_DP=<number of GCDs>. TP=2 across the two GCDs of one MI250 / MI250X: AEON_TP=2 AEON_DEQUANT_AT_LOAD=1 AEON_DFLASH="$AEON_DIR/dflash2" AEON_MAX_MODEL_LEN=131072 AEON_MAX_NUM_SEQS=16 (xGMI; no NCCL overrides).
• AITER stays off (vLLM's AITER needs CDNA 3 or newer).

Not supported

• Single GPUs under 28 GB: RX 7900 XTX (24 GB), RX 7900 XT (20 GB), every 16 GB card (RX 9070 XT / 9070 / 9060 XT 16 GB, RX 7900 GRE, RX 7800 XT, RX 7700, Radeon PRO W7700) and 12 GB cards (RX 9070 GRE, RX 7700 XT). About 20.76 GiB of text-only weights (measured on one R9700) leaves no usable KV cache at 90% utilization; even 28–32 GB single cards only run eager at 16K context (measured on R9700; 32K estimated). Smallest working combinations: 2x 24 GB (E), 2x 20 GB (M), 2x 16 GB (C for RDNA 4, L for RDNA 3), 4x 12 GB (D variant).
• 8 GB cards (RX 9060 XT 8 GB, RX 7600 and similar): even four of them leave no room for activations and cache.
• Architectures ROCm supports but the vLLM ROCm image isn't built for: Instinct MI100 (gfx908) and RDNA 2 (gfx1030: Radeon PRO W6800 / V620, RX 6800–6950 XT). vLLM's ROCm build targets gfx90a, gfx942, gfx950, gfx1100, gfx1101, gfx1150, gfx1151, gfx1200 and gfx1201. Also gfx1102 (RX 7600 / 7600 XT, Radeon PRO W7600 / W7500), which is in neither AMD's ROCm Linux list nor the image, and the MI50 / MI60 (gfx906), which current ROCm no longer supports.
• gfx1250: vLLM ships a separate image for it; these recipes don't cover it yet.
• Integrated GPUs other than Strix Halo (Ryzen AI 300 / gfx1150, Radeon 780M / gfx1103 and older): too little memory and bandwidth.
• TP=3 and TP=8: the model has 4 KV heads, which must split evenly. Use TP=1, 2 or 4, or replicas with data parallelism.
• Virtual machines with GPU passthrough for multi-GPU Radeon recipes: RCCL initialization has been reported to hang. Single-GPU recipes (B, F, G, P) don't use RCCL; for two cards, use bare metal or Recipe Z.
• Windows and WSL: not covered; vLLM's ROCm images are Linux-only.
• NVIDIA GPUs: use NVFP4-MIXED.
• llama.cpp, Ollama, LM Studio: this is not a GGUF.

Vision (Instinct and RDNA 3 only)

Every recipe starts text-only, which leaves the most memory for context. On RDNA 4 the vision tower currently fails to load in vLLM (vllm#49851), so keep --language-model-only there. Elsewhere, to serve images:

1. Set AEON_VISION=1 with the script, or edit the docker run:
   • remove --language-model-only;
   • add --limit-mm-per-prompt '{"image":1,"video":0}' --mm-processor-kwargs '{"max_pixels":1048576}';
   • on Radeon / Strix Halo, add -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE before the image name.
2. The vision tower adds about 0.9 GB, split across the cards.

The image-size cap matters. Without it, vLLM sizes the encoder's startup memory test for this processor's maximum of about 16.7 megapixels (about 16K tokens per image), which can run out of memory. With the cap it plans for about 1,024 tokens per image. Vision is untested with this checkpoint.

Troubleshooting

The full symptom → cause → fix table is in AGENTS.md. The most common ones:

Expected performance (estimates, not measurements)

The first AMD measurements are in Measured on 2x R9700: they cover the stock path, Recipe B, A-Max and a community-built optimized runtime, not yet the kit's patched recipes below. The 2x R9700 rows are calibrated against those measurements (below); a bandwidth-only per-pass model predicted the stock path well (about 1.9 tok/s for A-plain and 5 for A-Max against 1.7 and 5.5 measured) but was about 1.7x optimistic for the patched path. The figures below combine measurements of this exact checkpoint in vLLM (MXFP6 emulated; an equivalent W4A4 MLP path with the same MXFP4 weight grid and the same activation quantize-dequantize, which differs from vLLM's emulation only in GEMM accumulation order, so it is not bit-identical; one request, 384-token outputs; see Quality and validation status) with a memory-bandwidth scaling to each AMD system. Treat every figure as roughly ±50%.

How the estimates are worked out:

• Per forward pass. In the reference run, stock MXFP6 emulation took about 2.1 s per pass (12–20 ms per Gated DeltaNet projection call, 144 calls), MXFP4 emulation about 0.32 s, and the BF16 rest about 0.04 s, for about 2.47 s in total. With the MXFP6 weights dequantized once, the MXFP6 part drops to the 0.044 s the same layers cost in BF16 (about 0.40 s per pass); with the MXFP4 MLP also in BF16 after load, about 0.21 s. Each AMD estimate scales that cost by memory bandwidth, divides by the TP size, and adds about 8 ms of PCIe all-reduce at TP=2 (about 12 ms at TP=4). On 2x R9700 that bandwidth model gives about 94 ms per pass (about 10.6 tok/s) with the patch and about 0.53 s (about 1.9 tok/s) without it. Calibration (2x R9700, community-measured): stock decodes at about 1.7 tok/s (about 590 ms per token); unpacking MXFP6 costs about 8.7 ms per Gated DeltaNet layer, about 420 ms over 48 layers; so about 170 ms per pass is everything else (MXFP4 emulation, the BF16 layers, activation quantize-dequantize, all-reduce), which the patch doesn't touch. The R9700 estimates above use that 170 ms (about 6 tok/s plain), not the model's 94 ms.
• Speculation. Speedup = mean acceptance length ÷ (cost of one draft-plus-verify step ÷ cost of one plain step). Under emulation the verify step costs little more than a plain one, because the weight dequantization dominates and doesn't depend on how many tokens are checked; the step ratio is about 1.14 for DFlash2 and about 1.2 for MTP. So DFlash2 ≈ 3.73 ÷ 1.14 ≈ 3.3x plain, MTP ≈ 2.3x. When a pass gets cheap (both BF16 copies, many GPUs) the drafter's own cost weighs more and the gain shrinks.
• Acceptance on this target. The drafter accepts about 5–7% less on this MXFP4/MXFP6 checkpoint than on the NVFP4 build it was fine-tuned against (3.73 vs 4.01 at T=0.6, same prompts); at 4 and 8 concurrent requests the two were on par.
• Concurrency. Weight traffic is per forward pass, not per request, so aggregate throughput rises with concurrency until compute or memory runs out. 16 concurrent requests are unmeasured; the tester matrix measures them.
• Not modelled: RDNA 4 kernel efficiency, HIP-graph capture of the Quark ops, and RCCL behaviour with NCCL_P2P_DISABLE=1. Read the A-plain time per token first; it calibrates everything else.

Other systems (single stream, default recipe path; estimates ±50%). These still come from the bandwidth model alone; on 2x R9700 it was about 1.7x optimistic for the patched ("once") path, so read the patched figures below as the top of their range:

At Instinct bandwidths (about 5.3 TB/s for MI300X / MI300A, 8 TB/s for MI355X) kernel-launch overhead, not memory bandwidth, likely sets the single-stream limit, so no figure is given; MI210 / MI250 figures are optimistic for the same reason.

For scale only: Puget Systems reports 15.9 tok/s stock and 62.8 tok/s tuned with MTP at concurrency 1 on 2x R9700 with Qwen3.6-27B in FP8 (article). That is a different model and a different format with no emulation.

Testers: the step-by-step tester kit ships with the files, in TESTER_GUIDE.md, tests/run_radeon_eval.py and scripts/ (start with scripts/preflight_smoke.sh). Agents: AGENTS.md.

Expected behavior and limits

• Speed. Under emulation, decode speed is limited by how many bytes each forward pass moves, not by the 23.7 GB of packed weights. See Expected performance. Tensor parallelism, speculative decoding (DFlash2, MTP) and concurrent requests all spread that cost, and the MXFP6 dequant-once patch (plus MXFP4 dequant-at-load on the largest GPUs) removes most of it wherever it fits, without changing the output. Prefill is affected much less. On MI350/MI355 the MLP can also run natively (opt-in, not bit-identical).
• Memory.
  • Weights (measured by the community tester): 20.76 GiB text-only at TP=1, 10.51 GiB per card at TP=2. The vision tower adds about 0.86 GiB and the MTP head 0.79 GiB (0.40 per card at TP=2, measured). The MXFP6 dequant-once patch adds about 6.3 GiB net in total (+3.19 per card measured at TP=2 in the community runtime), MXFP4 dequant-at-load 23.8 GiB (+11.9 per card measured at TP=2), the DFlash2 drafter 3.58 GiB; each divided by the TP size.
  • Graphs (measured): 2.0–2.5 GiB per GPU with stock (pure-torch) per-pass emulation (2.50 at TP=1, 2.01 at TP=2); 1.3–1.4 GiB with the MLP packed as W4A4 in the community runtime (fused MXFP6 unpack per pass, or MXFP6 at load); none with --enforce-eager.
  • Activations and profiling (derived from the measured logs at 4,096 batched tokens, not logged directly): about 5.2 GiB at TP=1 with graphs, 2.2 GiB at TP=1 eager, 2.1–2.8 GiB per GPU at TP=2 with graphs; add an estimated 15–30% at 8,192 batched tokens.
  • Pool = VRAM × --gpu-memory-utilization − weights − graphs − activations. Both community-measured start failures (Recipe B's and A-Max's first defaults) sat at the low end of that range, so the defaults are now sized to the low end.
  • KV cache: 64 KiB per token at BF16 (16 full-attention layers × 4 KV heads × 256 × 2 × 2 bytes), divided by the TP size; 17 layers (+6%) with MTP. Every recipe uses BF16 KV, the reference numerics; FP8 KV (32 KiB) is an opt-in that changes outputs. vLLM allocates in 784-token blocks on this model.
  • Gated DeltaNet state: about 147 MiB per state page, FP32 as the model config sets it, divided by the TP size. A running sequence holds 2 + K pages with prefix caching on (vLLM's default for this model, "align" mode; K = draft tokens, so 2 without speculation), and 1 + K without prefix caching. This fits two community-measured logs exactly. At TP=2 that is about 0.14 GiB per GPU for A-plain and about 0.36 GiB with MTP K=3 (AGENTS.md section 12.2 has the formula).
  • Context length is per sequence: what 262K means. The GPU KV cache size and Maximum concurrency lines in the startup log are the real capacity.
• DFlash2 is untested on AMD; MTP (K=3) has run there in the community 2x R9700 test (A-Max and the optimized runtime; Measured). Measured τ for this checkpoint: see Quality and validation status. DFlash2: see Recipe A. MTP: see Recipe A-MTP; use {"method":"mtp",...} (the older name qwen3_5_mtp still works but logs a deprecation warning). Always add "attention_backend":"TRITON_ATTN" to the speculative config.
• Thinking.
  • Thinking is on by default, and the chat template's default reasoning effort is xhigh.
  • For quick answers, send "chat_template_kwargs": {"enable_thinking": false}.
  • For short thinking, send {"enable_thinking": true, "reasoning_effort": "low"}.
  • Set both through chat_template_kwargs, not through a top-level request field.
• Inherited behavior. The BF16 master is an Early Access Draft, and very long answers can fall into loops. 4-bit MLP weights, including the residual-writing down_proj, may make that more likely. The tester kit checks for loops explicitly.
• Uncensored. This model writes what the base model refuses. Read User responsibility.
• Runtimes.
  • vLLM is the only supported path.
  • This is not a GGUF, so llama.cpp, Ollama and LM Studio can't load it.
  • Transformers with amd-quark may load it in emulation, but that is untested.
  • Apple needs an MLX conversion.
• Known ROCm issues.
  • Decode speed on the R9700 can land about 20% lower for the life of a process (legacy-rocm-build#6347); restart and re-measure.
  • A TP=2 hang on dual R9700 is still reported open (vllm#40980); Recipes A, A-MTP and A-plain use the settings of the working community setups (NCCL_PROTO=Simple, plus NCCL_P2P_DISABLE=1), and Recipe Z avoids inter-GPU communication entirely.
  • vLLM v0.31.0 (ROCm 7.2.3 inside the image) has now run at TP=2 on 2x R9700 with this checkpoint in the community test (host ROCm 7.2.4; Measured); v0.28.0 uses the same ROCm 7.2.3 base and stays the escape hatch for vLLM-level errors.
  • vLLM v0.28–v0.30 can't load this checkpoint as shipped (algo_config crash in the Quark loader); use v0.31.0, or the AEON_STRIP_ALGO_CONFIG=1 escape hatch.
  • With this hybrid model, DFlash2 speculative decoding needs prefix caching off on vLLM 0.29–0.31 (vllm#55601, vllm#58894).
  • The first-start Gated DeltaNet compile hang on RDNA 4 (vllm#45929) is fixed in the images these recipes use.

Quality and validation status

Acceptance and quality were measured on a reference run of this exact checkpoint; the first AMD runtime results come from the community 2x R9700 test (Measured), and the full AMD quality score is pending.

Quantization recipe

• Tool: AMD Quark 0.12.post1 in eager mode, with Transformers 5.14.1 and PyTorch 2.13. The export is real_quantized: packed uint8 weights plus uint8 E8M0 scales. The Quark MX settings are scale_calculation_mode: even and round_method: half_even.
• Algorithm: AutoSmoothQuant with an MSE scale search, applied to the MLP only, along the edges post_attention_layernorm → gate/up and up_proj → down_proj. The smoothing factors are folded into those norms and weights. The GDN projections have no safe smoothing edge, so they get none.
• Calibration: 1,024 chat, code and math samples (UltraChat, Open-Platypus, CodeAlpaca, GSM8K), each truncated to 64 tokens, for 65,536 tokens in total. To fit in memory, the AutoSmoothQuant scale search used a 64-token subsample per layer. MX weight scales come from the weights themselves and activations are quantized at runtime, so calibration data only affects the MLP smoothing factors.
• Exact config: see quantization_config in config.json and BAKE_STAMP_Q2.json.

Model family

User responsibility

By accessing, downloading or running this model, you agree to the following:

1. You are responsible for its use. You alone are responsible for every prompt, every response, every downstream action, and any harm that results.
2. No warranty. The model is provided "AS IS", without warranty of any kind.
3. Follow the law. You must comply with all applicable laws and policies in every jurisdiction you operate in.
4. Add safety layers in production. Use input validation, output filtering, access controls, and human review for high-risk workflows.
5. The duty of care is yours. An uncensored model doesn't refuse on your behalf, so if you are unsure about a request, don't make it.
6. No endorsement. The authors do not endorse any particular output.
7. Arbitration. Disputes go to binding individual arbitration (under the AAA Consumer Rules if no other body is agreed), waiving jury trials and class actions.
8. Indemnification. You indemnify the authors, contributors and publishers against claims arising from your use.
9. Severability. If a provision is invalid, the closest enforceable equivalent replaces it.
10. Acceptance. Using the model means you accept these terms. If you don't accept them, don't use the model.

Support the work

AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development:

Become a member on Patreon → patreon.com/cw/AeonForge7/membership

Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Supporters also get early access to new releases, such as the AEON DFlash2 drafter.

License and attribution

• License: Apache-2.0, inherited from Qwen/Qwen3.8-27B.
• Qwen team (Alibaba): the Qwen3.8-27B base model.
• AEON-7: the uncensored fine-tune (the BF16 master) and this quantization. The upstream tools behind the fine-tune are credited on the BF16 card.
• AMD Quark: the quantization toolkit used to produce the OCP MX export.