Phi-3.5-mini-instruct-AWQ-INT4

INT4 weight-only quantization of microsoft/Phi-3.5-mini-instruct.

Microsoft Phi-3.5-mini-instruct in INT4. About 2.5 GB on disk. Runs on a 4 GB consumer GPU.

Property Value
Base model microsoft/Phi-3.5-mini-instruct
Quantization INT4 weight-only
Approx. on-disk size ~2.3 GB
License MIT License
Languages English

Load (vLLM)

vllm serve drawais/Phi-3.5-mini-instruct-AWQ-INT4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.94

from vllm import LLM, SamplingParams
llm = LLM(model="drawais/Phi-3.5-mini-instruct-AWQ-INT4", max_model_len=32768)
print(llm.generate(["Hello!"], SamplingParams(max_tokens=128))[0].outputs[0].text)

Footprint

~2.3 GB on disk. Recommended VRAM: enough headroom for KV cache.

License & attribution

This artifact is a derivative work of microsoft/Phi-3.5-mini-instruct, released by its original authors under the MIT License.

This artifact is distributed under the same license. The full license text is included in LICENSE, and required attribution is in NOTICE.

License text: https://opensource.org/license/mit Source model: https://huggingface.co/microsoft/Phi-3.5-mini-instruct