XYZ-Aquila-mini NVFP4 (SM121)
True W4A4 NVFP4 (ModelOpt 0.45) of
XYZAILab/XYZ-Aquila-mini
@ 4d3b0c45663b8b40f9e6e57b64ff815c09c49904, served natively on NVIDIA GB10 / SM121
with vLLM 0.25.0.
Native model context: max_position_embeddings = 262144 (~256k).
Served window depends on host KV (max_model_len); NIAH uses a max-context protocol
(25/50/90% of served max_model_len).
Runtime
docker pull ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814
docker run --gpus all --ipc=host --network host \
-e PORT=18082 \
-e MODEL_ID=/models/model \
-e SERVED_MODEL_NAME=xyz-aquila-nvfp4 \
-e QUANTIZATION=modelopt_mixed \
-e KV_CACHE_DTYPE=fp8 \
-e ATTENTION_BACKEND=flashinfer \
-e MOE_BACKEND=flashinfer_b12x \
-e LINEAR_BACKEND=flashinfer_cutlass \
-e LANGUAGE_MODEL_ONLY=1 \
-e GPU_MEMORY_UTILIZATION=0.85 \
-e MAX_MODEL_LEN=32768 \
-e MAX_NUM_SEQS=16 \
-v $PWD:/models/model:ro \
ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814
Long-context tip: raise MAX_MODEL_LEN toward 262144 with MAX_NUM_SEQS=1 and
chunked prefill (MAX_NUM_BATCHED_TOKENS can stay 16384). Host-admit before claiming full native ctx.
Native markers: FlashInferCutlassNvFp4LinearKernel, FLASHINFER_B12X MoE, FlashInfer attn arch=sm121, FP8 KV. No Marlin/emulation.
Quant recipe
- Eligible MLP/experts/
lm_head: NVFP4 W4A4 - Attention: FP8
- KV cache at serve: FP8
- Excluded: vision, routers, norms, embeddings, GDN aux, MTP (MTP off in derived config only)
- Calibration: 512 windowed rows
- Export: ~21 GiB, 3 shards; hf_quant 291 layers (NVFP4 161 / FP8 130)
Quality results
| Suite | Score | Notes |
|---|---|---|
BFCL v4 multi_turn_base (200) |
64.00% campaign · 68.00% core-subset re-run | bfcl-eval==2025.12.17 · 0 infra errors |
| BFCL AST-600 micro | 33.33% prior · 33.50% core-subset | multiple / parallel / parallel_multiple |
| AST multiple | 35.0% / 35.5% | 200 cases |
| AST parallel | 46.5% / 48.5% | 200 cases |
| AST parallel_multiple | 18.5% / 16.5% | 200 cases |
| API canaries | 5/5 PASS | identity, zh math, JSON, tool, needle |
| NIAH MAX_CONTEXT_TEST (served M=native 262144) | 3/3 PASS | depths 65,520 / 131,040 / 235,872 (25/50/90% of M−64) |
| NIAH (legacy served M=32768) | 3/3 PASS | ~8k/16k/29k — not full native capacity |
Do not use diluted BFCL “Overall Acc” CSV columns when only a subset of categories was run.
Throughput (GB10, vllm bench serve)
1024 in / 256 out, ignore_eos, temp 0, 3 reps drop-first:
| Conc. | Output tok/s | Total tok/s | mean TTFT ms | mean TPOT ms | Failed |
|---|---|---|---|---|---|
| 1 | 64.6 | 325.4 | 694 | 12.8 | 0 |
| 2 | 82.4 | 415.3 | 2010 | 16.3 | 0 |
| 4 | 164.5 | 829.1 | 641 | 21.9 | 0 |
| 8 | 230.5 | 1162.1 | 1009 | 30.9 | 0 |
| 16 | 305.7 | 1540.9 | 1854 | 45.3 | 0 |
Conversion gates
| Gate | Status |
|---|---|
| B1 BF16 CPU real-residency baseline (~70.2 GiB) | PASS |
| C W4A4 PTQ | PASS |
| D packed NVFP4 export audit | PASS |
| E SM121 native serve | PASS |
Related
- Runtime repo: https://github.com/r0b0tlab/xyz-aquila-mini-nvfp4-sm121-vllm
- Bench client: https://github.com/r0b0tlab/r0b0bench (
core/core-subsetprofiles) - Evidence HTML: see runtime repo
docs/evidence-report.html
License
Weights follow upstream XYZ-Aquila-mini / Apache-2.0 where applicable. Quantization & packaging © r0b0tlab.