Hy3 295B-A21B — PrismaQuant 5.3-bit mixed NVFP4/FP8/BF16 for 2× DGX Spark
Tencent Hy3 (July 2026 production release, Apache 2.0) — 295B total /
21B active MoE, 80 layers, 192 routed experts top-8, including the MTP
draft layer for speculative decoding — quantized by
PrismaQuant from the BF16 source
to a ~190 GB compressed-tensors checkpoint (5.28 bits/param over
quantizable weights) sized to serve on two NVIDIA DGX Sparks under
vLLM pipeline parallelism with no custom kernels and no forked runtime.
Sibling artifact: the single-Spark 2.8-bit GGUF is at
rdtand/Hy3-295B-A21B-PrismaQuant-2.8bit-gguf-vllm.
What's inside
- Measured mixed-precision allocation (per-tensor formats from a
knapsack solve over measured per-(tensor, format) error weighted by an
empirical Fisher probe of the BF16 model): NVFP4 on the routed-expert
bulk, FP8 (E4M3, dynamic activations) on most attention/shared
tensors, BF16 on the small sensitive remainder. Full map in
allocation/layer_config.json. - Byte-budget selection: sized so weights fit 2×128 GB with KV headroom (~99 GB/node under pp2), not tuned to a curve heuristic.
- Deliberate rendering throughout: GPTQ + static activation ordering + joint scale optimization on dense Linears; batched GPTQ on every one of the 15,360 routed experts (rendered inline during the streaming export — the model never fits in memory at any point in the pipeline).
- MTP included:
model.layers.80ships verbatim BF16 and loads into vLLM's HYV3 MTP path (--speculative-config '{"method":"mtp",...}'). - Embeddings/head BF16; norms/router FP32.
Validation — read this
No quality claims are made for this artifact, and the author has one Spark: the full 2-node topology has not been run. What was validated:
- A 6-layer truncated twin of this exact artifact (same quantization metadata, real tensors, MTP remapped) passes vLLM load + generate and MTP speculative-decode smokes on a single Spark. pp2 loads each rank through the same unsharded per-layer path the twin exercises.
- A static tp2 shard audit (shipped:
serving/tp2_shard_audit.py) verifies every quantized tensor's packed dims, NVFP4 group-16 boundaries, and scale shapes divide cleanly at world size 2. - Per-tensor packing is bit-exact against the pipeline's own emulation.
Community measurements (2×Spark throughput, KL/perplexity vs BF16, task suites) are welcome and will be linked here.
Serving
See serving/README.md for the two-node pp2 launch (recommended) and the
tp2 alternative.
Provenance
Quantized with PrismaQuant. Source: tencent/Hy3 (BF16). Allocation and
Pareto data in allocation/. Contact: robert.tand@icloud.com