Index-Translate-2B mobile GGUF
Text-only GGUF quantizations of IndexTeam/Index-Translate-2B for on-device translation with llama.cpp (Android/iOS). The vision tower is not included. Every file keeps the 1-layer MTP block, so llama.cpp's MTP draft mode works if you want it (it did not speed up CPU decoding in our tests).
Which file
| file | size | quality vs BF16 (COMET / chrF++) | use |
|---|---|---|---|
Index-Translate-2B-Q4_K_M.gguf |
1.31 GB | -0.15 / -0.25 | recommended default: practically lossless, all constraint checks pass |
Index-Translate-2B-Q2_K-QAT.gguf |
0.78 GB | -2.2 / -3.0 | low-RAM or speed-critical devices; about 1.5x faster decode than Q4_K_M on CPU |
Index-Translate-2B-Q3_K_M.gguf |
1.13 GB | -0.8 / -1.2 | middle ground |
Index-Translate-2B-Q4_0.gguf |
1.24 GB | -0.3 / -0.6 | when your build's ARM repacking is fastest for Q4_0 |
Index-Translate-2B-Q6_K.gguf |
1.61 GB | ~0 | near-reference |
Index-Translate-2B-Q8_0.gguf |
2.08 GB | ~0 | reference-quality |
The Q6_K, Q4_K_M, Q4_0 and Q3_K_M files use an importance matrix computed on about 1,400 translation prompts in the official prompt format (FLORES-200 dev sentences, 52 languages, no overlap with the test set). The output head (tied with the embeddings) stays at q6_K in Q4_K_M / Q6_K and q8_0 in Q8_0.
Q2_K-QAT comes from quantization-aware training rather than plain rounding. All 24 transformer layers were trained against the exact llama.cpp Q2_K format (per-block 4-bit scales and mins, 2-bit weights, straight-through estimator, learned scales), with KL distillation from the BF16 model's full-vocabulary logits. Training data was 250k translation prompts in the official prompt formats (FLORES-200 dev and OPUS-100 validation sentences, 52 languages, plain, constrained and JSON variants), answered greedily by the BF16 model. That is about 38M tokens, one epoch, and the embeddings were kept frozen. The trained weights convert to Q2_K bit-exactly (except fp16 scale rounding on 0.002% of weights). The MTP block and output head are q4_K, the DeltaNet alpha/beta projections are q8_0. For comparison, plain post-training Q2_K at the same layout and size (with imatrix) scores 81.3 COMET / 42.7 chrF++, and IQ2_M (0.90 GB) scores 83.5 / 45.7. Q2_K-QAT gets 85.8 / 47.7.
Known weaknesses of Q2_K-QAT: lower-resource and Indic/Thai targets lose the most (en-hi, en-bn, en-th about -5
COMET). In one of the 12 constraint checks it translated {name}/{count} placeholders. If your app relies on
placeholders or glossary terms, use Q4_K_M or protect the placeholders before translating.
Prompt format
Use the chat template shipped in the GGUF, with thinking disabled (the template's empty <think> block), and greedy
decoding. Plain translation prompt (as used by the original model):
请将以下{源语言}文本翻译为{目标语言}，直接输出翻译结果，不要进行任何解释。
{text}
Example: llama-cli -m Index-Translate-2B-Q4_K_M.gguf -t 4 --temp 0 -cnv
Evaluation
FLORES-200 devtest, first 200 sentences per direction, 20 directions. Greedy decoding with the same prompts for every variant, run with llama.cpp. COMET is Unbabel/wmt22-comet-da. chrF++ uses word bigrams, which penalizes Chinese and Japanese targets because they have no spaces, so character-only chrF is shown as well. BF16 is the unquantized GGUF. Its outputs match the original transformers model (COMET 87.98, chrF++ 50.77 under the same setup). Constraint checks are 12 prompts covering glossary terms, JSON value-only translation, placeholders and markdown.
COMET-22 (wmt22-comet-da, x100)
| pair | BF16 | Q8_0 | Q6_K | Q4_K_M | Q4_0 | Q3_K_M | Q2_K-QAT |
|---|---|---|---|---|---|---|---|
| en-zh | 89.5 | 89.5 | 89.5 | 89.5 | 89.3 | 89.2 | 88.7 |
| zh-en | 87.7 | 87.7 | 87.6 | 87.7 | 87.4 | 87.4 | 86.9 |
| en-ja | 92.0 | 91.9 | 91.9 | 91.9 | 91.8 | 91.8 | 91.0 |
| ja-en | 88.4 | 88.4 | 88.3 | 88.4 | 88.3 | 88.1 | 87.4 |
| en-ko | 90.6 | 90.7 | 90.7 | 90.6 | 90.5 | 90.5 | 89.0 |
| en-de | 88.4 | 88.5 | 88.4 | 88.6 | 88.3 | 88.2 | 87.0 |
| en-fr | 89.3 | 89.3 | 89.3 | 89.2 | 89.2 | 89.1 | 88.0 |
| en-es | 87.7 | 87.7 | 87.6 | 87.8 | 87.7 | 87.3 | 86.9 |
| en-ar | 88.3 | 88.3 | 88.1 | 88.1 | 87.9 | 87.8 | 86.8 |
| ar-en | 87.1 | 87.1 | 87.1 | 86.9 | 87.0 | 86.6 | 85.6 |
| en-hi | 82.2 | 82.2 | 82.3 | 82.0 | 82.0 | 81.2 | 76.4 |
| en-th | 90.0 | 90.0 | 90.1 | 90.0 | 89.6 | 89.4 | 85.4 |
| en-vi | 90.5 | 90.4 | 90.4 | 90.3 | 90.3 | 90.1 | 89.5 |
| en-id | 92.4 | 92.4 | 92.5 | 92.3 | 92.3 | 91.9 | 91.3 |
| en-ru | 90.2 | 90.1 | 90.2 | 90.1 | 90.2 | 89.6 | 88.4 |
| en-uk | 89.8 | 89.8 | 89.8 | 89.5 | 89.5 | 89.2 | 86.3 |
| zh-ja | 91.1 | 91.2 | 91.3 | 91.1 | 91.1 | 90.8 | 90.3 |
| en-sw | 80.1 | 80.2 | 79.9 | 79.2 | 78.3 | 77.1 | 77.1 |
| sw-en | 78.7 | 78.7 | 78.6 | 78.2 | 77.3 | 74.1 | 75.1 |
| en-bn | 85.9 | 86.0 | 85.7 | 85.6 | 85.2 | 84.0 | 79.1 |
| avg | 87.99 | 88.00 | 87.97 | 87.84 | 87.66 | 87.17 | 85.82 |
chrF++
| pair | BF16 | Q8_0 | Q6_K | Q4_K_M | Q4_0 | Q3_K_M | Q2_K-QAT |
|---|---|---|---|---|---|---|---|
| en-zh | 29.1 | 29.1 | 28.2 | 28.9 | 28.2 | 29.8 | 26.5 |
| zh-en | 56.5 | 56.5 | 56.6 | 56.5 | 56.0 | 55.8 | 54.5 |
| en-ja | 32.6 | 32.5 | 36.4 | 36.0 | 31.9 | 35.0 | 30.5 |
| ja-en | 53.7 | 53.8 | 53.5 | 53.8 | 53.3 | 52.5 | 51.3 |
| en-ko | 36.8 | 36.7 | 36.7 | 36.2 | 36.1 | 35.9 | 34.2 |
| en-de | 61.0 | 61.2 | 61.0 | 61.3 | 60.7 | 59.9 | 58.3 |
| en-fr | 69.6 | 69.6 | 69.6 | 68.4 | 69.0 | 68.5 | 67.4 |
| en-es | 54.5 | 54.6 | 54.5 | 54.9 | 54.1 | 54.3 | 53.1 |
| en-ar | 54.1 | 53.9 | 53.9 | 53.0 | 52.7 | 52.4 | 50.9 |
| ar-en | 61.2 | 61.1 | 61.2 | 60.3 | 60.8 | 59.9 | 58.0 |
| en-hi | 53.5 | 53.6 | 53.5 | 53.3 | 53.0 | 51.8 | 47.2 |
| en-th | 47.1 | 47.1 | 47.1 | 46.8 | 47.1 | 46.6 | 42.6 |
| en-vi | 61.7 | 61.6 | 61.2 | 61.1 | 61.5 | 60.4 | 59.4 |
| en-id | 68.2 | 68.1 | 68.4 | 67.8 | 68.3 | 67.0 | 65.4 |
| en-ru | 55.7 | 55.4 | 55.6 | 55.4 | 55.2 | 53.5 | 52.5 |
| en-uk | 51.0 | 50.7 | 50.7 | 50.4 | 49.9 | 49.8 | 48.2 |
| zh-ja | 28.8 | 28.9 | 28.8 | 28.5 | 28.8 | 28.2 | 26.4 |
| en-sw | 50.5 | 50.5 | 50.2 | 48.8 | 48.1 | 47.1 | 47.8 |
| sw-en | 50.6 | 50.8 | 51.2 | 50.3 | 49.9 | 46.4 | 46.7 |
| en-bn | 39.1 | 39.2 | 39.0 | 38.4 | 38.3 | 36.9 | 33.9 |
| avg | 50.76 | 50.75 | 50.87 | 50.51 | 50.15 | 49.57 | 47.74 |
chrF (character n-grams only)
| pair | BF16 | Q8_0 | Q6_K | Q4_K_M | Q4_0 | Q3_K_M | Q2_K-QAT |
|---|---|---|---|---|---|---|---|
| en-zh | 37.3 | 37.3 | 36.8 | 37.0 | 36.9 | 37.5 | 34.8 |
| zh-en | 59.1 | 59.1 | 59.2 | 59.1 | 58.6 | 58.4 | 57.1 |
| en-ja | 43.1 | 43.0 | 43.3 | 42.8 | 42.3 | 41.2 | 37.9 |
| ja-en | 56.6 | 56.7 | 56.4 | 56.7 | 56.2 | 55.5 | 54.1 |
| en-ko | 38.0 | 38.0 | 37.9 | 37.4 | 37.5 | 37.1 | 35.5 |
| en-de | 63.5 | 63.6 | 63.5 | 63.8 | 63.3 | 62.5 | 61.1 |
| en-fr | 71.4 | 71.3 | 71.4 | 70.2 | 70.7 | 70.3 | 69.3 |
| en-es | 56.8 | 56.9 | 56.8 | 57.2 | 56.4 | 56.6 | 55.6 |
| en-ar | 57.9 | 57.8 | 57.7 | 56.9 | 56.6 | 56.2 | 54.7 |
| ar-en | 63.3 | 63.3 | 63.4 | 62.5 | 63.0 | 62.0 | 60.2 |
| en-hi | 55.6 | 55.8 | 55.6 | 55.5 | 55.2 | 53.9 | 49.4 |
| en-th | 57.0 | 56.9 | 57.0 | 56.6 | 56.9 | 56.0 | 51.3 |
| en-vi | 61.7 | 61.6 | 61.3 | 61.1 | 61.5 | 60.4 | 59.5 |
| en-id | 70.4 | 70.3 | 70.6 | 70.1 | 70.6 | 69.2 | 67.6 |
| en-ru | 58.5 | 58.1 | 58.3 | 58.1 | 57.9 | 56.2 | 55.4 |
| en-uk | 54.3 | 54.0 | 54.0 | 53.7 | 53.2 | 53.0 | 51.4 |
| zh-ja | 35.5 | 35.5 | 35.5 | 35.3 | 35.4 | 35.0 | 32.7 |
| en-sw | 53.3 | 53.3 | 53.0 | 51.5 | 51.0 | 50.0 | 50.5 |
| sw-en | 52.1 | 52.3 | 52.7 | 51.9 | 51.5 | 47.8 | 48.0 |
| en-bn | 44.0 | 44.1 | 43.9 | 43.3 | 43.2 | 41.7 | 38.6 |
| avg | 54.47 | 54.45 | 54.42 | 54.03 | 53.88 | 53.02 | 51.23 |
| constraint checks (12) | 12 | 11 | 11 | 12 | 11 | 11 | 9 |
Runaway generations (hit the 1,024-token limit) out of 4,012 prompts: BF16 0, Q8_0 0, Q6_K 1, Q4_K_M 2, Q4_0 1, Q3_K_M 0, Q2_K-QAT 5 (IQ2_M 22, plain Q2_K 40). Set a max-token limit proportional to input length in apps.
Size and speed
Decode on a phone is mostly limited by memory bandwidth. "Bytes per token" counts the weights read for each generated token: the full body plus the full 248k-row output head. The MTP block is excluded.
| file | file size | bytes / token | estimate at 50 GB/s | measured, 16-vCPU EPYC VM, 4 threads | measured, Apple M4 Max CPU, 4 threads |
|---|---|---|---|---|---|
| Q8_0 | 2.08 GB | 2.00 GB | 25 tok/s | - | - |
| Q6_K | 1.61 GB | 1.55 GB | 32 tok/s | 16.3 tok/s | - |
| Q4_K_M | 1.31 GB | 1.26 GB | 40 tok/s | 17.3 tok/s | 12.1 tok/s |
| Q4_0 | 1.24 GB | 1.19 GB | 42 tok/s | - | - |
| Q3_K_M | 1.13 GB | 1.09 GB | 46 tok/s | - | - |
| Q2_K-QAT | 0.78 GB | 0.74 GB | 68 tok/s | 26.2 tok/s | 16.6 tok/s |
The estimates are upper bounds. Measured CPU numbers are lower because the Gated DeltaNet recurrence and K-quant dequantization are partly compute-bound at 4 threads. The VM numbers are medians of 3 interleaved rounds on a shared host and have roughly ±10% noise. Compare them within a column, not across columns.
The output head accounts for 33% (Q4_K_M) to 39% (Q2_K-QAT) of the bytes per token. Restricting the head to the tokens a target language can use (an experimental runtime patch that computes logits only for a token subset) measured +34% decode speed for Q4_K_M and +44% for Q2_K-QAT with a 16k-token subset on the same VM. It needs a modified runtime, and it changes 0-2% of outputs for non-Latin targets, so it is not part of these files.
Acknowledgements
Huge thanks to Hugging Face for generously sponsoring the compute used for this project 🤗
License
Apache-2.0, same as the base model.