This is a quantized version of thinkingmachines/Inkling with MoE layers quantized to NVFP4 and attention layers quantized to FP8 Block.
This model is intended for deployment with vLLM. You can serve the model using 4xB200s:
vllm serve RedHatAI/Inkling-NVFP4-FP8-BLOCK \
--tokenizer-mode inkling \
--tensor-parallel-size 4 \
--enable-auto-tool-choice \
--tool-call-parser inkling \
--reasoning-parser inkling \
--enable-expert-parallel \
--max-model-len 100000 \
--max-num-seqs 32 \
--enable-chunked-prefill \
--trust-remote-code \
Evaluations were performed with inspectai:
inspect eval hf/Idavidrein/gpqa/diamond \
--model RedHatAI/Inkling-NVFP4-FP8-BLOCK \
--reasoning-effort high \
--model-base-url http://localhost:8000/v1 \
| Benchmark | thinkingmachines/Inkling |
RedHatAI/Inkling-NVFP4-FP8-BLOCK |
|---|---|---|
| GPQA Diamond | 87.2 | 84.5 |
Note: A bug in inspectai currently misscores markdown-formatted multiple choice answers (e.g. ANSWER: $B$ rather than ANSWER: B), which the Inkling models often include in their response. Scores are manually evaluated to account for discrepancy. More information at https://github.com/UKGovernmentBEIS/inspect_ai/issues/5145
model-00001-of-00032.safetensors 20 GBmodel-00032-of-00032.safetensors 20 GBmodel-00002-of-00032.safetensors 17 GBmodel-00003-of-00032.safetensors 17 GBmodel-00031-of-00032.safetensors 17 GBmodel-00011-of-00032.safetensors 17 GBmodel-00005-of-00032.safetensors 17 GBmodel-00016-of-00032.safetensors 17 GBmodel-00006-of-00032.safetensors 17 GBmodel-00028-of-00032.safetensors 17 GBmodel-00008-of-00032.safetensors 17 GBmodel-00009-of-00032.safetensors 17 GBmodel-00012-of-00032.safetensors 17 GBmodel-00013-of-00032.safetensors 17 GBmodel-00015-of-00032.safetensors 17 GBmodel-00018-of-00032.safetensors 17 GBmodel-00019-of-00032.safetensors 17 GBmodel-00021-of-00032.safetensors 17 GBmodel-00023-of-00032.safetensors 17 GBmodel-00025-of-00032.safetensors 17 GBmodel-00026-of-00032.safetensors 17 GBmodel-00029-of-00032.safetensors 17 GBmodel-00022-of-00032.safetensors 17 GBmodel-00027-of-00032.safetensors 17 GBmodel-00004-of-00032.safetensors 17 GBmodel-00017-of-00032.safetensors 17 GBmodel-00007-of-00032.safetensors 17 GBmodel-00010-of-00032.safetensors 17 GBmodel-00014-of-00032.safetensors 17 GBmodel-00020-of-00032.safetensors 17 GBmodel-00024-of-00032.safetensors 17 GBmodel-00030-of-00032.safetensors 17 GBmodel_mtp.safetensors 11 GB有文件?提交种子。
提交种子thinkingmachines/Inkling