Hy-MT2-30B-A3B NVFP4 Experts GGUF

An expert-only NVFP4 derivative of tencent/Hy-MT2-30B-A3B for native English-Persian and Persian-English translation on NVIDIA Blackwell GPUs.

This is not a blanket 4-bit conversion. The 128 routed experts in layers 1-47 are NVFP4; attention, embeddings, routers, shared experts, layer 0, and the language-model head remain BF16. The routed experts account for 94.4198% of the 30,064,719,872 source parameters.

Provenance

Source: tencent/Hy-MT2-30B-A3B
Source revision: d3ead4dba61c09aac60a261a96ad1df3e705febb
Quantizer: NVIDIA ModelOpt 0.45.0 at ec87a82927d003986d44fb7f4fa8b3d10c31b095
Runtime/converter: llama.cpp at ebb546b7e961bd46fd9ed0387ffd14ca86b6fe1b
Calibration: 240 English/Persian Wikipedia samples, 65,487 tokens, collected on 2026-08-11; benchmark sources and references were excluded
Artifact size: 19,353,433,056 bytes
SHA-256: 3f5edeb940b56ad851cce84cc55de7812d97dc80aeb4a6d96aa20f044bec891f

ModelOpt fake-quant validation on eight held-out calibration rows produced 0.979527 prefill-logit cosine similarity and 0.973255 first-decode-logit cosine similarity, with 8/8 top-1 agreement in both cases.

Validation

The native GGUF was tested on an RTX PRO 6000 Blackwell Server Edition with a 64-row balanced development set (32 translations in each direction). References are synthetic development evidence, not an independent human test set.

Runtime	Format	Output tok/s	en-fa chrF++	fa-en chrF++
vLLM	BF16 source	859.40	56.86	66.00
llama.cpp	NVFP4 GGUF	413.92	56.84	65.49

Both runs used batch/concurrency 64 and a maximum of 128 generated tokens. The NVFP4 run produced no empty translations and no exact source copies. Its output agreement with BF16 was 88.01 chrF++ for English-to-Persian and 93.66 chrF++ for Persian-to-English.

RTX 5080 Laptop (Stallion)

The same GGUF and llama.cpp revision were also validated on a 16 GB RTX 5080 Laptop GPU at an 85 W power limit. With 38 layers offloaded to CUDA and the remaining weights in system RAM, a single 8K-context slot used 15,144 MiB VRAM and generated at 53.9 tok/s server-side (51.9 tok/s end-to-end).

An eight-slot throughput run used 8K total context (1,024 tokens per slot), 64 requests, and a 128-token output cap:

Concurrency	Output tok/s	Total tok/s	Requests/s
8	85.63	231.87	1.69

That run produced no empty outputs or exact source copies. Development chrF++ was 56.17 for English-to-Persian and 65.80 for Persian-to-English.

llama.cpp

Use a CUDA build that includes SM120 and the NVFP4 MMQ kernel. The exact tested llama.cpp revision is listed above.

cmake -S . -B build-sm120 \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build-sm120 --config Release -j --target llama-cli llama-server

build-sm120/bin/llama-cli \
  -m Hy-MT2-30B-A3B-NVFP4-Experts.gguf \
  -ngl 99 -c 4096 -fa on \
  -n 256 --temp 0

The amount of GPU offload is hardware- and workload-dependent. Leave headroom for the KV cache and other processes instead of assuming the complete 19.35 GB file fits in VRAM.

The tested 16 GB / 8K Stallion configuration used -ngl 38 -c 8192 -np 1 for single-request serving. For the aggregate batch-8 result it used -ngl 38 -c 8192 -np 8; llama.cpp therefore assigned 1,024 context tokens to each parallel slot.

Important limitation

The intermediate ModelOpt safetensors pass fake-quant logit checks, but their current native and emulated vLLM deployment paths generated invalid text during this build. Those files are quarantined and are not the supported artifact. The GGUF listed by the exact checksum above is the validated runtime artifact.

The development set was used to validate conversion fidelity. It must not be reported as a fresh held-out measure of general translation quality.

Licensing and attribution

This model distribution is licensed under the Apache License, Version 2.0. See LICENSE. Existing third-party copyright, license, and attribution notices remain applicable.

Upstream: tencent/Hy-MT2-30B-A3B; declared license: apache-2.0.
- Retained notice: LICENSE.txt.