Mistral-7B-Instruct-v0.3-SpinQuant-NVFP4
INT4 weight-only quantization of mistralai/Mistral-7B-Instruct-v0.3.
Mistral-7B-Instruct-v0.3 in NVFP4 W4A4 (4-bit weights + 4-bit activations). Native vLLM compressed-tensors. Hopper/Ada/Blackwell tensor cores supported. About 4 GB on disk.
| Property | Value |
|---|---|
| Base model | mistralai/Mistral-7B-Instruct-v0.3 |
| Quantization | INT4 weight-only |
| Approx. on-disk size | ~4.5 GB |
| License | Apache License, Version 2.0 |
| Languages | English |
Load (vLLM)
vllm serve drawais/Mistral-7B-Instruct-v0.3-SpinQuant-NVFP4 \
--max-model-len 32768 \
--gpu-memory-utilization 0.94
from vllm import LLM, SamplingParams
llm = LLM(model="drawais/Mistral-7B-Instruct-v0.3-SpinQuant-NVFP4", max_model_len=32768)
print(llm.generate(["Hello!"], SamplingParams(max_tokens=128))[0].outputs[0].text)
Footprint
~4.5 GB on disk. Recommended VRAM: enough headroom for KV cache.
License & attribution
This artifact is a derivative work of mistralai/Mistral-7B-Instruct-v0.3,
released by its original authors under the Apache License, Version 2.0.
This artifact is distributed under the same license. The full license text is
included in LICENSE, and required attribution is in NOTICE.
License text: https://www.apache.org/licenses/LICENSE-2.0 Source model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3