Qwen3.6-35B-A3B-heretic — NVFP4 (v2 multimodal-preserved)

✅ Validated on the unified AEON vLLM Ultimate image ghcr.io/aeon-7/aeon-vllm-ultimate:latest (= :2026-06-18-v0.23.0-dflashfix; rollback :2026-06-11-pr41703) — vLLM 0.23.0 built from source for GB10 / sm_121a + the AEON DFlash stack. Loads + serves cleanly (compressed-tensors MoE) — including working multimodal image understanding (a vision tensor-prefix bug was fixed 2026-06-18 — see the Vision fix section below) — and with DFlash num_speculative_tokens: 11 now scales to c=64 with up to ~740 tok/s aggregate and no concurrency crash. This single image supersedes the older vllm-spark-omni-q36:v1.2 container. Recommended container base.

🚀 PRODUCTION DEPLOYMENT GUIDE: github.com/AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4-DFlash

The GitHub repo is the definitive turn-key setup for DGX Spark — end-to-end deployment guide, validated OpenClaw config, and a concurrency-sweep benchmark harness.

• Image: ghcr.io/aeon-7/aeon-vllm-ultimate:latest (vLLM 0.23.0 source-built for sm_121a + the AEON speculative-decoding stack — see What we fixed for the DGX Spark)
• Pairs with z-lab/Qwen3.6-35B-A3B-DFlash drafter (must be post 2026-04-19 revision)
• Production-stable under sustained chat load and concurrent agent fleets up to c=64 on DGX Spark

What changed in v2 (2026-04-19)

v1 of this checkpoint had model.language_model.layers.X.* keys remapped to model.layers.X.* so vLLM's text-only Qwen3_5MoeForCausalLM loader would pick them up. That layout was unstable in production — intermittent NaN/crash in the prefix-strip codepath during real chat sessions.

v2 re-quantizes the same source (tvall43/Qwen3.6-35B-A3B-heretic) with AutoModelForImageTextToText, restoring the canonical multimodal architecture (the text path):

• Architecture: Qwen3_5MoeForConditionalGeneration (vLLM's canonical class — no registry hack required)
• Keys: model.language_model.layers.X.* retained natively (no post-quantization key rewriting)
• 27-block ViT vision encoder preserved BF16 — but note its tensors were mis-nested as model.language_model.visual.* and silently skip-loaded until the 2026-06-18 vision fix below; image inputs did not work in v2 until that rename
• 30 linear-attention (Mamba/GDN) layers preserved BF16
• All 122,880 per-expert NVFP4 keys (40 layers × 256 experts × 3 projections × 4 quant components)

vLLM serves it via the canonical multimodal class with no prefix-strip code path in the inference hot loop. Result: rock-solid text stability where v1 was crashing on virtually every interaction. (Vision, however, was not yet functional in v2 — the ViT was mis-nested and skip-loaded; see the Vision fix section below, which corrected this on 2026-06-18.)

⚠️ If you cloned v1 of this repo, delete and re-pull. Same URL — v2 commits replaced v1.

Vision fix (2026-06-18) — image inputs now work

v2 (above) preserved the vision tower's BF16 weights, but a quantization-time ignore-regex defect nested the 27-block ViT one level too deep: the 333 vision tensors were written as model.language_model.visual.* instead of the sibling model.visual.* that vLLM's Qwen3_5MoeForConditionalGeneration expects (it builds self.visual at model.visual — a sibling of self.language_model, not a child). vLLM therefore silently skip-loaded the entire vision tower (WARNING … Parameter visual.* not found in params_dict, skip loading) and ran it uninitialized — text was perfect, but any image input produced !!!! garbage.

This revision renames those 333 vision tensors to model.visual.* — a header-only edit, so the NVFP4/BF16 weight data is byte-for-byte identical (no re-quantization; same file size, same data offsets). vLLM now loads the full ViT (0 skip-loads) and image understanding works, validated on aeon-vllm-ultimate:latest: a shapes/colors probe scored 7/7 (all shapes + colors + on-image text) and a richer scene 5/7, with LLM text unchanged. The AEON-7 27B multimodal siblings already used the correct model.visual.* layout and were unaffected by this defect.

If you pulled this repo before 2026-06-18 and saw !!!! on image prompts, re-pull — the corrected model.safetensors is live at the same URL (weight data identical; only the vision tensor names changed).

---

NVFP4-quantized version of tvall43/Qwen3.6-35B-A3B-heretic — an abliterated (decensored, 5/100 refusal rate) Qwen 3.6 35B-A3B Mixture-of-Experts multimodal model with thinking/reasoning capabilities.

Quantized using llmcompressor with the compressed-tensors nvfp4-pack-quantized format. Calibrated with 256 samples from open-platypus over 40 sequential decoder-layer stages. Vision encoder, linear-attention (Mamba/GDN) layers, MoE routers, gates, norms, and lm_head/embed_tokens preserved in BF16.

Designed for deployment on NVIDIA DGX Spark (GB10, Blackwell SM 12.0+) with native FP4 tensor-core support. Pairs with z-lab/Qwen3.6-35B-A3B-DFlash for spec-decode acceptance of 2.7-4.4 mean accepted tokens per target step on greedy workloads.

---

🚀 Quickstart (copy-paste)

Complete recipe for a DGX Spark / Blackwell box: pull the container, pull this NVFP4 model, pull the DFlash drafter (fresh), then serve with the vetted flags + DFlash spec-decode. No --mamba-block-size is needed for this 35B-A3B body.

# 1. Pull the unified AEON vLLM image (anonymous public GHCR pull)
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest

# 2. Pull THIS NVFP4 model (fresh — v2 commits replaced v1 at the same URL)
huggingface-cli download AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4 --local-dir ./aeon-model

# 3. Pull the DFlash drafter, fresh (must be a post 2026-04-19 revision)
huggingface-cli download z-lab/Qwen3.6-35B-A3B-DFlash --local-dir ./aeon-drafter

# 4. Serve (NVFP4 body + DFlash drafter, n=11)
docker run --gpus all --ipc host --network host \
  -e VLLM_TEST_FORCE_FP8_MARLIN=1 \
  -v ./aeon-model:/model:ro \
  -v ./aeon-drafter:/drafter:ro \
  --entrypoint vllm \
  ghcr.io/aeon-7/aeon-vllm-ultimate:latest \
  serve /model \
  --quantization compressed-tensors \
  --attention-backend flash_attn \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --gpu-memory-utilization 0.65 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --trust-remote-code \
  --max-model-len 40960 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 32768 \
  --speculative-config '{"method":"dflash","model":"/drafter","num_speculative_tokens":11}'

Required with DFlash: --max-num-seqs and --max-num-batched-tokens must be set (as above) — without them the speculative-decoding scheduler can compute a negative token budget and the server fails to boot (max_num_scheduled_tokens is set to -512…). Raise --max-model-len (e.g. 262144) for full long-context once running; see the production compose below.

Lower --gpu-memory-utilization to 0.70 if you co-locate ASR/TTS/embedding services on the Spark's unified memory — on GB10 keep it 0.6-0.7, since above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box (even 0.85 stalls); go lower for co-located services, high concurrency, fp16 KV, or DFlash. For the full production setup (docker-compose, served-model aliases, 256K --max-model-len, systemd, OpenClaw integration) see the Quick Start (DGX Spark) and Production docker-compose sections below.

---

Performance Benchmarks

v0.23.0 build — current production image

These are the current measured numbers on ghcr.io/aeon-7/aeon-vllm-ultimate:latest (vLLM 0.23.0 + AEON sm_121a build + DFlash num_speculative_tokens: 11 via z-lab/Qwen3.6-35B-A3B-DFlash), on NVIDIA DGX Spark (GB10, sm_121a, 128 GB unified memory).

The unified image's specific win on this 35B-A3B MoE is clean concurrency scaling to c=64 with no crash (the pre-fix image crashed under concurrent speculative decoding at c≥32 — see What we fixed for the DGX Spark), plus long-context DFlash acceptance that holds ~42–58% out to 32K-token prompts. Aggregate throughput rises from ~75–124 tok/s single-stream to ~430–740 tok/s at c=64 depending on category — DFlash-acceptance-driven: ~610–740 tok/s on structured code/math/reasoning/extraction (which draft well) and ~430–475 tok/s on creative prose/natural-language (lower draft acceptance).

Single-stream (c=1), by category

Greedy (T=0), 256-token output, mixed-domain prompt set. decode_tok_s_p50 / ttft_p50_ms / tpot_p50_ms / prefill (pp_tok_s_p50) / DFlash position-0 acceptance.

Aggregate throughput vs concurrency (c=1 → c=64)

Aggregate tok/s across all active streams (median of the per-category levels), 256-token output:

Scales cleanly through c=64 with zero errors at every level (the pre-fix image died at c≥32 under speculative decoding). Aggregate is compute-bound on this 35B-active-3B MoE — per-request decode falls as streams divide the GPU, but no stream stalls or crashes.

Re-validated 2026-06-19 on the exact published recipe (a fresh pull of aeon-vllm-ultimate:latest + the corrected weights, DFlash n=11): multimodal vision works end-to-end (7/7 on an image probe, 0 vision skip-loads) and the throughput spread reproduced with zero errors at c=64 — structured categories 705–781 tok/s @ c=64 (peak ~800 tok/s @ c=32, Reasoning), creative-text 430–474 tok/s @ c=64; DFlash acceptance ~46–55% (structured) / ~22–27% (creative) at short context, holding ~40–45% at 16–20K-token context.

Long-context tiers (DFlash acceptance holds)

Measured at ~16K and ~32K input tokens (median prompt_tokens_p50), greedy, 256-token output. The point of the AEON DFlash patches (PR #40898 SWA + PR #41703 prefix-cache safety) is that draft acceptance does not collapse as context grows:

Acceptance at 32K (43–58%) is on par with — and on the Extraction/JSON class, above — short-context acceptance, confirming the drafter's sliding-window layers are running as true SWA rather than collapsing past ~2K tokens.

Stock baseline note: there is no stock vanilla-vLLM baseline for this 35B-A3B checkpoint yet. A fresh fully-vanilla re-benchmark (default vLLM, no DFlash, no AEON/sm_121a optimizations) is pending; when it lands, a stock-vs-optimized contrast will be added here. The figures above are all measured on the optimized aeon-vllm-ultimate:latest (vLLM 0.23.0) build. The earlier v1.2-image figures retained below for reference are likewise optimized-path numbers (different image, DFlash k=15), not a vanilla baseline.

⚠️ DFlash speedup is workload-dependent. Greedy reasoning workloads (math, code, extraction) hit the highest acceptance (≥40%); creative / open-ended / sampled prose is lower (~24%) and more variable. Use T=0 for maximum DFlash speedup.

---

Earlier vllm-spark-omni-q36:v1.2 reference (DFlash k=15)

The detailed single-stream / concurrent tables below were measured on the earlier ghcr.io/aeon-7/vllm-spark-omni-q36:v1.2 image at DFlash k=15, not on the current aeon-vllm-ultimate:latest at n=11. They are retained as a representative reference for the Spark DFlash path; the current production numbers are the v0.23.0 section above. The unified image's specific gain is clean c=64 concurrency scaling and held long-context acceptance, not these short-context single-stream tok/s.

1. Single-Stream Performance

Best for interactive chat and agentic UX. All measurements greedy (T=0) unless noted.

Decode rate (10 trials, 200-token outputs)

Variance reflects DFlash acceptance differences across prompt classes — math/code prompts hit ~125 tok/s with high drafter agreement, more open-ended prompts settle around 60-90 tok/s.

TTFT by prompt length (5 trials per class)

Sub-130ms TTFT for any prompt under ~50 tokens — fixed kernel-launch overhead dominates short prefill.

Decode rate by output length (3 trials per length)

* model emitted EOS naturally before hitting max_tokens.

Decode rate increases with output length — DFlash steady-state amortization improves over the first 100-200 tokens once the drafter and target lock into a stable acceptance pattern.

Sampling: greedy vs stochastic (5 trials per mode)

15% degradation T=0 → T=0.7. Less dramatic than typical for spec-decode systems — DFlash's drafter remains useful even at moderate sampling. Use T=0 for max DFlash speedup; T=0.7 for diversity.

Long-prompt prefill (RAG / document workloads)

Prefill rate plateaus around 2K tok/s due to (a) the drafter prefilling the same context in parallel and (b) Qwen3.6's 30 linear-attention (Mamba/GDN) layers having higher prefill constant factor than parallel softmax attention. Decode-after-prefill drops gracefully (~50% from 1K → 32K context).

Single-stream summary

---

2. Concurrent-Session Performance

Best for agent fleets and multi-user serving. 3 trials per level, median run reported (sorted by aggregate throughput). Mixed prompts, 200-token output, T=0.7 (stochastic — production-realistic), SSE streaming.

Throughput scaling (N concurrent clients, 200-tok output)

Zero errors across all 384 requests in the concurrent sweep (3 runs × 128-conc top level alone = 384, plus all lower levels = 1,200+ total).

Aggregate throughput plateaus at ~313 tok/s from 64 concurrent onward — that's the GPU's compute wall on this 35B-active-3B MoE with linear-attention KV reads + DFlash drafter overhead. TTFT spikes severely at 128 concurrent (14s p50, 47s p95) because all 128 sequences fit in the scheduler but compute is fully saturated, so each token's worth of work is divided across 128 streams. For latency-sensitive UX, target 16-32 concurrent; for max throughput, use the full 128.

TTFT-only scaling (1-token output, prefill + first-token)

Measures pure scheduler queue contention — critical for agent UX:

TTFT stays sub-700ms through 64 concurrent — smooth UX for small agent fleets. Beyond 64, TTFT accumulates queue-wait time as compute is fully consumed.

Concurrent with 1K-token prompts (RAG-style workload)

50-token output with 1,024-token prompts — simulates agents doing document QA or retrieval-augmented responses. Median of 2 runs.

RAG throughput peaks around 50 tok/s at 16-64 concurrent. The aggregate is lower than the short-prompt sweep because each request spends most of its wall-clock in prefill (1K tokens) rather than decode. Use prefix caching if your RAG workload has repeated context blocks — the production compose enables --enable-prefix-caching which can give 5-10× speedup on shared-prefix RAG.

Concurrent-session summary

---

Key Performance Metrics Summary

---

Scaling efficiency (200-tok concurrent test)

Scaling is GPU-compute-bound rather than memory-bound — DFlash on a 35B MoE with hybrid linear+full attention saturates the GB10's compute around 64 concurrent. Per-request throughput degrades from 109 tok/s (1-req) to 6.5 tok/s (128-req). For comparison, a non-spec-decode setup would scale much more linearly but lose the ~2-4× single-stream speedup DFlash provides.

---

Test methodology notes

• enable_thinking=false — bench disables Qwen3.6's thinking tag for clean decode-rate measurement. Production with thinking on adds reasoning-token overhead before content emission (use max_tokens ≥ 2048 for thinking-enabled requests).
• DFlash speedup is workload-dependent — math, code, agentic, and reasoning workloads at T=0 hit the highest acceptance rates. Creative writing or open-ended chat sees lower acceptance.
• Mixed-prompt set in concurrent tests: code, math, QA, creative writing, single-line answers — to avoid biasing toward DFlash-friendly prompts.
• 3 trials per concurrency level for the throughput sweep, median run (by aggregate tok/s) reported. RAG section uses 2 trials.
• 200-token output as the standard test length (except TTFT-only test which uses 1 token, RAG which uses 50, and decode-by-output which sweeps 50→1000).
• Error tracking: 0/1,200+ requests failed across the full test (all sections combined).
• Reproducible: bench script at scripts/bench_full.py; raw JSON results at bench/qwen36_v2_2026-04-20.json.

---

⚠️ IMPORTANT REQUIREMENTS

---

Quick Start (DGX Spark, with DFlash spec decode)

# 1. Pull the image (anonymous public GHCR pull — anyone can run this)
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest

# 2. Pull both models
sudo mkdir -p /opt/qwen36 && sudo chown $USER:$USER /opt/qwen36
cd /opt/qwen36
export HF_HUB_ENABLE_HF_TRANSFER=1
hf download AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4 --local-dir ./qwen36-nvfp4 &
hf download z-lab/Qwen3.6-35B-A3B-DFlash         --local-dir ./qwen36-dflash &
wait

# 3. Get the production compose file
curl -fsSL \
  https://raw.githubusercontent.com/AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4-DFlash/main/examples/docker-compose.yml \
  -o docker-compose.yml

# 4. Start
docker compose up -d
docker compose logs -f   # wait for "Application startup complete" (~3-5 min)

# 5. Test (use temperature=0 + ≥2048 max_tokens for thinking-enabled requests)
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen36-fast",
    "messages": [{"role":"user","content":"What is 17 × 23? Show your work."}],
    "max_tokens": 2048,
    "temperature": 0
  }'

Full step-by-step (with pre-flight checks, smoke tests, systemd service, OpenClaw integration): github.com/AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4-DFlash/blob/main/docs/dgx-spark-setup.md

Production docker-compose (the actual flags that work)

services:
  vllm:
    image: ghcr.io/aeon-7/aeon-vllm-ultimate:latest   # = :2026-06-18-v0.23.0-dflashfix; rollback :2026-06-11-pr41703
    container_name: vllm-qwen36-heretic
    restart: unless-stopped
    network_mode: host
    # The image ENTRYPOINT is /bin/bash, so use entrypoint: vllm + command: serve ...
    entrypoint: vllm
    environment:
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - TORCH_CUDA_ARCH_LIST=12.1a
      - ENABLE_NVFP4_SM100=0
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - NVIDIA_FORWARD_COMPAT=1
      - VLLM_TEST_FORCE_FP8_MARLIN=1     # pins the NVFP4 MoE backend to MARLIN (256×512 expert shape)
    volumes:
      - /opt/qwen36/qwen36-nvfp4:/models/qwen36
      - /opt/qwen36/qwen36-dflash:/models/qwen36-dflash
    command:
      - serve
      - /models/qwen36
      - --served-model-name
      - qwen36-35b-heretic
      - qwen36-fast
      - qwen36-deep
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --tensor-parallel-size
      - "1"
      - --dtype
      - auto
      - --quantization
      - compressed-tensors
      - --max-model-len
      - "262144"
      - --max-num-seqs
      - "64"
      - --max-num-batched-tokens
      - "32768"
      - --gpu-memory-utilization
      - "0.70"
      - --enable-chunked-prefill
      - --enable-prefix-caching
      - --trust-remote-code
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_coder
      - --reasoning-parser
      - qwen3
      - --speculative-config
      - '{"method":"dflash","model":"/models/qwen36-dflash","num_speculative_tokens":11}'
      - --attention-backend
      - flash_attn
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Recipe notes (current build):

• --quantization compressed-tensors (NOT modelopt) — this checkpoint uses llmcompressor's NVFP4 format.
• --attention-backend flash_attn is required for the DFlash drafter on the body.
• num_speculative_tokens: 11 is the validated production default for this 35B-A3B body — acceptance and long-context hold peak around n≈10–11, above which they fall off. Do not set --kv-cache-dtype (the non-causal DFlash drafter requires BF16 KV).
• No --mamba-block-size is needed on the v0.23.0 image (block_size is now an int upstream; the prior patch_kv_cache_utils carry was dropped).
• --enforce-eager is NOT required — CUDA graphs run cleanly with the unified image's spec-decode CUDA-graph capture-size alignment patch + the post-2026-04-19 DFlash drafter. Cudagraphs on give ~30% over eager.
• --gpu-memory-utilization 0.70 leaves headroom for co-located ASR/TTS/embedding services on the Spark's unified memory; keep it 0.6-0.7 even when the LLM is the only major GPU service — above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box (even 0.85 stalls). Go lower for co-located services, high concurrency, fp16 KV, or DFlash.

---

What we fixed for the DGX Spark

All AEON models run on one unified container — ghcr.io/aeon-7/aeon-vllm-ultimate:latest (= :2026-06-18-v0.23.0-dflashfix; rollback :2026-06-11-pr41703) — vLLM v0.23.0 built from source for GB10 / sm_121a and merged with the AEON speculative-decoding stack.

The result

• Scales to 64 concurrent requests with no crash — the prior image crashed at c≥32 under speculative decoding (the block-table fix above). Aggregate throughput rises to ~430–740 tok/s at c=64 depending on category (acceptance-driven: ~610–740 on structured text, ~430–475 on creative prose/natural-language), zero errors at every level.
• Native NVFP4 4-bit compute on Blackwell tensor cores — the speed of 4-bit with near-16-bit accuracy.
• Speculative decoding (DFlash) holds high draft acceptance from short prompts (32–48% short-context) through long 16K–32K agent histories (43–58% at 32K), rather than collapsing past ~2K tokens.
• No measured single-stream speedup vs a stock vanilla baseline yet — there is no stock baseline for this checkpoint; a fresh fully-vanilla re-benchmark is pending. The unified image's concrete, measured win here is the c=64 concurrency scaling and the held long-context acceptance described above.

---

Model Architecture

Hybrid Attention

---

Quantization Details

Quantized layers (NVFP4)

• Gated Attention projections: q_proj, k_proj, v_proj, o_proj (10 layers)
• MoE experts (256 × 40 layers = 10,240 expert modules): gate_proj, up_proj, down_proj
• Shared expert: same projections

Excluded from quantization (kept BF16)

• lm_head, embed_tokens — accuracy-critical token projections
• *.mlp.gate, *.shared_expert_gate — MoE routing (sparsity-critical)
• *.norm.* — all RMSNorm layers
• *.visual.* — 27-block ViT vision tower
• *.linear_attn.* — 30 Gated DeltaNet (Mamba) layers (small relative to MoE; quantizing them tanks accuracy)

The exact recipe + script that produced this checkpoint is at scripts/qwen36_requant_v2.py.

---

Recommended sampling parameters

From the Qwen3.6 model card:

For maximum DFlash speedup: use T=0 (greedy). The drafter ↔ target agreement rate collapses with sampling — at T=0.7 you typically see ~10-20% acceptance vs. 60-78% at T=0.

The production compose registers 3 served-model aliases for the same backend so chat clients can route greedy vs sampled requests separately:

• qwen36-fast → intended for greedy/agentic (T=0)
• qwen36-deep → intended for creative/sampled (T=0.7)
• qwen36-35b-heretic → canonical name

Disable thinking per-request:

{"chat_template_kwargs": {"enable_thinking": false}}

Preserve thinking across multi-turn:

{"chat_template_kwargs": {"preserve_thinking": true}}

Common gotcha: with thinking enabled (default), Qwen3.6 spends most of its max_tokens budget on <think> reasoning before emitting content. Use max_tokens ≥ 2048 for thinking-enabled requests — lower budgets often produce content: null with finish_reason: "length".

---

Hardware Requirements

---

Files

---

Disclaimer

THIS IS AN UNCENSORED MODEL. By downloading, accessing, or using this model you expressly assume full and sole responsibility for all outputs generated, all actions taken based on outputs, and compliance with applicable laws. The authors are not responsible for any harmful, illegal, or objectionable content. These tools serve legitimate purposes including security research, red-teaming, content analysis, and creative work. Implement safeguards appropriate to your use case and jurisdiction.

License

Apache 2.0 (inherited from Qwen3.6 base).

Credits

• Base model: tvall43/Qwen3.6-35B-A3B-heretic — abliteration via Heretic v1.2.0
• Original target: Qwen/Qwen3.6-35B-A3B by Alibaba Tongyi
• DFlash drafter: z-lab/Qwen3.6-35B-A3B-DFlash — z-lab (Soroush Mohri et al.)
• Quantization tool: llmcompressor — Neural Magic / RedHat
• vLLM build chain: vllm-project/vllm v0.23.0 source-built for sm_121a + the AEON speculative-decoding stack (aeon-vllm-ultimate)
• SM121 stability investigation: rmagur1203/vllm-dgx-spark — independent 4-day, 144-config investigation that surfaced the Marlin requirement
• Quantized by: AEON-7 on NVIDIA RTX PRO 6000 Blackwell (RunPod)
• Production-validated on: NVIDIA DGX Spark (GB10)

---

☕ Support the work

AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development.

Patreon wrongfully shut down our account without notice and never responded to our appeals, so we built something even better. You can now support the work on LotusMake: buy us a Lotus Tea, fund GPU hours, back a project you want to see done, or subscribe for early and full access to publications while supporting what we build. You can become a creator there too: join, apply to become a creator, and get paid for the work you do in public.

AEON FORGE on LotusMake → lotusmake.com/c/aeon-forge · Subscribe for early and full access

Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Subscribers also get early access to new releases, such as the AEON DFlash2 drafter.

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.