Multilingual Quality Benchmark: MiniMax H3 Text Encoder
4-bit (nvfp4_awq) vs FP8
Tested 13 languages on nvfp4_awq and found:
- Short prompts (everyday use): the 4-bit encoder is basically perfect.
- Long, detailed prompts (the kind MiniMax H3 is designed for): about
2 out of every 100 words get their meaning slightly scrambled. In a
prompt with 1000+ words, that's a handful of words the model misunderstands
- which is exactly where "wrong pronunciation", "broken sound on a word", or "the character does the wrong thing" come from.
- Some languages are hit worse than others. Greek, Hindi and Mandarin were the worst (3+ in 100 words), Turkish the best (1.5 in 100). German was middle of the pack. English itself wasn't immune.
- The fix: an FP8 version of the encoder cuts the scrambled-word rate by more than half, and makes the overall meaning drift about half as strong. The words most at risk (rare vocabulary and non-English words) are kept in full precision.
Bottom line: if you generate in English with short prompts, your current 4-bit encoder is fine. If you use long, structured prompts or other languages, the FP8 encoder is a real quality upgrade for a modest file-size cost.
What the numbers mean
Every word in your prompt becomes a "meaning vector" inside the model. I compared the 4-bit encoder's vectors against the full-precision (BF16) encoder's vectors.
| Term | Plain meaning | Good value |
|---|---|---|
| Cosine similarity | How much a word's meaning matches the original. 1.0 = identical meaning; 0.9 = slightly different; 0.5 = meaning basically changed | 0.99+ = great |
| Outlier (a "scrambled word") | A word whose meaning the compressed encoder got wrong - below 0.95 similarity. In practice: that word generates with wrong pronunciation, broken sound, or a wrong action/object. | 0% is the goal |
| Outlier rate | How many scrambled words per 100 words in your prompt | as low as possible |
| RMS divergence | How much the whole prompt's meaning drifted from the original. High = even "correct" words feel slightly off (like a different tone or pace). | lower is better |
Real-world example: imagine a 500-word prompt. A 2% outlier rate means ~10 words the model partially misunderstands. If one of those words is "calm" or a character name, that's the broken pronunciation you noticed.
The test
- 13 languages: English, German, Spanish, French, Mandarin, Hindi, Russian, Arabic, Persian, Kurdish, Turkish, Greek, Hebrew
- 2 prompts per language: a long, technical MiniMax H3 prompt (the real usage) + a short everyday prompt
- Method: compare each compressed encoder against the full-precision encoder on the same text so only the compression's effect is measured.
Result 1 - the 4-bit encoder (nvfp4_awq), long prompts
| Lang | Scrambled words / 100 | Worst single word | Overall drift |
|---|---|---|---|
| EN | 1.8 | 0.61 | 9.4% |
| DE | 2.1 | 0.68 | 9.3% |
| ES | 2.6 | 0.59 | 12.4% |
| FR | 2.5 | 0.68 | 12.2% |
| ZH | 3.1 | 0.48 | 10.0% |
| HI | 3.3 | 0.55 | 11.1% |
| RU | 1.8 | 0.64 | 9.4% |
| AR | 2.0 | 0.57 | 14.0% |
| FA | 2.1 | 0.69 | 10.2% |
| KU | 2.2 | 0.67 | 12.6% |
| TR | 1.5 | 0.76 | 9.5% |
| EL | 3.4 | 0.52 | 12.2% |
| HE | 1.9 | 0.70 | 9.6% |
Short prompts: clean in all 13 languages (0 scrambled words).
What this means: long prompts are where the 4-bit encoder leaks. Greek, Hindi and Mandarin are the riskiest; Turkish held up best. English itself shows 1.8%, so no language is exempt. Arabic is different: its words were mostly fine, but the whole prompt drifted the most (14%) - the overall mood shifts even when individual words survive.
Result 2 - the FP8 encoder, the same long prompts
| Lang | Scrambled words / 100 | Before (4-bit) | Overall drift | Before |
|---|---|---|---|---|
| EN | 1.4 | 1.8 | 7.2% | 9.4% |
| DE | 0.9 | 2.1 | 4.8% | 9.3% |
| AR | 0.7 | 2.0 | 7.6% | 14.0% |
| EL | 1.3 | 3.4 | 5.7% | 12.2% |
| TR | 0.9 | 1.5 | 4.8% | 9.5% |
| ZH | 1.6 | 3.1 | 5.5% | 10.0% |
Short prompts: essentially lossless (0.999 similarity).
What this means: scrambled-word rate cut by more than half on average (up to 2.6× for Greek), meaning drift roughly halved, and short prompts now match the original almost perfectly. The remaining few outliers track structural/technical tokens present in every language's long prompt - not pure compression noise.
What was changed in the FP8 encoder
- All 467 linear (neural-network) weights -> FP8-E4M3, the compression level where word meanings survive
- Vocabulary embeddings, output head, positional data, normalization -> kept in full precision (BF16), this is where rare and non-English words live, so they stay untouched
Should you switch?
| Your usage | Recommendation |
|---|---|
| English, short prompts | 4-bit is fine, no need to change |
| English, long structured prompts | FP8 gives a real but modest gain (1.8 - 1.4%) |
| Any non-English, long prompts | FP8 is the clear upgrade (scrambled words roughly halved) |
| Maximum multilingual quality | BF16 (48 GB) remains the reference |
Content-behavior sanity check (refusal / harmful prompts)
I also ran a sanity check against the Semantic-Harmful dataset
(heretic-org/matched_harmful, 416 harmful prompts) to check content
behavior:
- This artifact is a text encoder, not a chat model. It has no refusal mechanism, its job is to map any prompt (including harmful ones) into conditioning vectors for the video model. Refusal is a text-generation behavior (the model saying "I can't help with that"), which does not apply to an encoder's function.
- The encoder processes all input prompts without filtering or refusal, and the quantized (FP8) version preserves encoding fidelity across content types, the same fidelity numbers shown in the multilingual results above apply regardless of prompt subject.