Granite-4.2-8B (NVFP4)
[!NOTE] This repository contains the NVFP4 version of Granite-4.2-8B.
Please refer to the original model's model card for additional details: https://huggingface.co/ibm-granite/granite-4.2-8b
[!NOTE] This repository contains the NVFP4 version of Granite-4.2-8B.
Please refer to the original model's model card for additional details: https://huggingface.co/ibm-granite/granite-4.2-8b
LocalMaxxingvllm serve ibm-granite/granite-4.2-8b-nvfp4 --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code在 LocalMaxxing 上查看测试
vllm serve ibm-granite/granite-4.2-8b-nvfp4 --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN在 LocalMaxxing 上查看测试
vllm serve ibm-granite/granite-4.2-8b-nvfp4 --max-model-len 24576 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code在 LocalMaxxing 上查看测试
来自 LocalMaxxing 的 16 次社区测试（3 种配置）的中位数。
自己运行：配置指南