中文Sign up

rdtand/Hy3-295B-A21B-PrismaQuant-2.8bit-gguf-vllm

🤗 Hugging Face sourceapache-2.021B activated104 GBGGUF✓ 3 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo rdtand/Hy3-295B-A21B-PrismaQuant-2.8bit-gguf-vllm ./model-folder
Needs a seeder →

Hy3 295B-A21B — PrismaQuant 2.8-bit GGUF (vLLM-servable)

Tencent Hy3 (the July 2026 production release, Apache 2.0 — not the preview) — 295B total / 21B active MoE, 80 layers, 192 routed experts top-8 — quantized by PrismaQuant from the BF16 source to a 103.7 GB GGUF (2.80 bits/param over quantizable weights) that serves on a single NVIDIA DGX Spark (128 GB unified memory) via vLLM + the GGUF plugin.

This supersedes the earlier Hy3-preview-based 2.9-bit artifact: production base model, Apache 2.0 license, and a quantization menu extended with the GGUF IQ formats (IQ2_XXS…IQ4_XS, E8-lattice codebooks) alongside the k-quants — the IQ rungs measured strictly better per byte than the k-rungs they displace at these bit-rates.

How it was made

  • Measured allocation: per-tensor formats chosen by a knapsack solve over measured per-(tensor, format) quantization error weighted by an empirical Fisher probe of the BF16 model (32×1024 calibration, streaming — the model never fits in memory), with per-format calibration factors derived from end-to-end KL measurements on a smaller proxy model.
  • Byte-budget selection: the bit-rate targets a 103.5 GB file (single-Spark serving with KV-cache headroom) rather than a curve heuristic. The measured rate-distortion knee sits at ~2.3 bpp — this artifact rides well above it.
  • Format mix: routed-expert bulk on IQ2_XS/IQ3_XXS/IQ4_XS; small attention/shared tensors on Q5_K/Q6_K/Q8_0; embeddings Q4_K; output head Q6_K; norms/router F32. Full map in allocation/layer_config.json.
  • imatrix-weighted quantization (per-column activation second moments from the same calibration corpus), exhaustive GPU codeword search for IQ formats.
  • The MTP layer (model.layers.80) is not included (GGUF serving does not use speculative decoding here).

Validation — read this

No quality claims are made for this artifact. Models of this scale cannot be KL-validated against their BF16 teacher on the target hardware. Validation performed: vLLM loads the artifact, greedy and chat generation are coherent across smoke prompts (math, prose, technical explanation), and the pipeline's per-tensor packing is bit-exact against its own emulation and gguf-py decoding. Community measurements welcome.

Serving (vLLM + GGUF plugin, single DGX Spark)

See serving/: serve.sh launches a vLLM container with the vllm-gguf-plugin and two small patches (container_patches.py fixes an upstream embed_tokens quant-config gap; hy_v3_gguf_adapter.py maps this artifact's HF-checkpoint-verbatim tensor names onto vLLM's Hy3 loader, including the stacked 192-expert MoE tensors). Note: all 2-D weight tensors in this artifact are quantized (no BF16 Linears) — required by the current plugin load path. llama.cpp does not implement this architecture at time of writing; serving is vLLM-only.

serving/serve.sh hy3-prismaquant-2.8bpp-00001-of-00003.gguf ./config

Provenance

Quantized with PrismaQuant (streaming direct-from-safetensors GGUF exporter). Source checkpoint: tencent/Hy3 (BF16). Allocation, Pareto sweep, and per-format calibration factors are in allocation/. Contact: robert.tand@icloud.com