Phi-4-multimodal-instruct W4A16 GPTQ

GPTQ W4A16 quantization of microsoft/Phi-4-multimodal-instruct — a 5.6 B parameter multimodal model by Microsoft supporting text, vision (images), and audio inputs.

Quantized with llm-compressor on RTX 5090. Weights stored in compressed-tensors format — natively loaded by vLLM.

License: MIT — © Microsoft Corporation. This quantization carries the same MIT license as the original model.

---

Why this quantization?

The GGUF files in Swicked86/phi4-mm-gguf are smaller but lack audio. This model is the sweet spot: all three modalities at roughly ⅓ the size of bf16.

---

Available Files

The SigLIP-400M vision encoder and conformer-based speech encoder are stored at full bfloat16 precision — only the Phi3 text transformer weights (32 decoder layers) are quantized to int4.

---

VRAM Requirements

Model weights occupy 6.11 GiB (measured). The remaining VRAM is used by the KV cache — vLLM pre-allocates the full KV cache pool at startup based on --gpu-memory-utilization and --max-model-len. Total VRAM allocated = weights + pre-allocated KV cache, regardless of how many requests are active.

By GPU tier

By context length

KV cache estimates use Phi-4-Mini architecture (32 layers, 8 KV heads, head_dim 96, bf16 activations ≈ 96 KB/token). Add ~1–2 GB for framework overhead. Weights measured on RTX 5090 with vLLM.

Why does vLLM show higher usage than "total est." above? vLLM pre-allocates the entire KV cache pool at startup. On a large GPU (e.g. 32 GB at --gpu-memory-utilization 0.45), it reserves 0.45 × 32 GB = ~14 GB for KV cache even if no requests are active. The table above shows the minimum needed, not what vLLM will allocate when given more headroom.

---

Usage

Step 1 — Install vLLM

Requirements: Python 3.10+, CUDA GPU with ≥ 8 GB VRAM, vLLM 0.9.0+

pip install vllm

Do not install auto-gptq or pass --quantization gptq. This model uses compressed-tensors format, which vLLM handles automatically from config.json.

---

Step 2 — Download the model

huggingface-cli download Swicked86/phi4-mm-gptq --local-dir ./phi4-mm-gptq

Or let vLLM download on first run by passing the repo ID directly (see Step 3).

---

Step 3 — Launch the vLLM server

Minimum working command (text + vision + audio, 8–10 GB GPU):

python -m vllm.entrypoints.openai.api_server \
  --model ./phi4-mm-gptq \
  --dtype bfloat16 \
  --trust-remote-code \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --enable-lora \
  --max-lora-rank 320 \
  --lora-modules speech=./phi4-mm-gptq/speech-lora \
                 vision=./phi4-mm-gptq/vision-lora \
  --limit-mm-per-prompt '{"image": 3, "audio": 3}' \
  --port 8080 \
  --host 127.0.0.1 \
  --served-model-name phi4-mm

Extended context command (16 GB+ GPU, 65K context):

python -m vllm.entrypoints.openai.api_server \
  --model ./phi4-mm-gptq \
  --dtype bfloat16 \
  --trust-remote-code \
  --max-model-len 65520 \
  --gpu-memory-utilization 0.90 \
  --enable-lora \
  --max-lora-rank 320 \
  --lora-modules speech=./phi4-mm-gptq/speech-lora \
                 vision=./phi4-mm-gptq/vision-lora \
  --limit-mm-per-prompt '{"image": 3, "audio": 3}' \
  --enable-auto-tool-choice \
  --tool-call-parser phi4_mini_json \
  --port 8080 \
  --host 127.0.0.1 \
  --served-model-name phi4-mm

Flag reference:

No --quantization flag needed. vLLM reads quantization_config from config.json and activates the compressed-tensors int4 kernels automatically.

Wait for "Application startup complete":

curl http://localhost:8080/health   # → {"status":"ok"}

---

Text (Python — openai SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=256,
)
print(response.choices[0].message.content)

---

Vision — image understanding (Python)

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
            {"type": "text", "text": "Describe this image in detail."},
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)

curl:

IMAGE_B64=$(base64 -w0 photo.jpg)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"phi4-mm\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/jpeg;base64,${IMAGE_B64}\"}},
        {\"type\": \"text\", \"text\": \"What is in this image?\"}
      ]
    }],
    \"max_tokens\": 300
  }"

---

Audio — speech transcription / understanding (Python)

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("audio.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
            {"type": "text", "text": "Transcribe this audio."},
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)

phi4-mm uses a custom conformer-based audio encoder (24 conformer blocks) with a rank-320 speech LoRA applied to the language decoder — no separate ASR model needed. Supported formats: wav, mp3, ogg, flac.

curl:

AUDIO_B64=$(base64 -w0 audio.wav)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"phi4-mm\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"input_audio\", \"input_audio\": {\"data\": \"${AUDIO_B64}\", \"format\": \"wav\"}},
        {\"type\": \"text\", \"text\": \"Transcribe and summarise.\"}
      ]
    }],
    \"max_tokens\": 512
  }"

---

Combined — image + audio in one prompt

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()
with open("question.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="phi4-mm",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url",  "image_url":  {"url": f"data:image/jpeg;base64,{image_b64}"}},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
            {"type": "text", "text": "Answer the spoken question about the image."},
        ],
    }],
    max_tokens=512,
)
print(response.choices[0].message.content)

---

Tool calling

phi4-mm emits tool calls as functools[{"name":"...","arguments":{...}}]. The --tool-call-parser phi4_mini_json flag (vLLM 0.7+) handles this automatically. For a complete chat template that injects tools into phi4-mm's native <|tool|>...<|/tool|> block, see deploy/wsl-vllm/phi4-mm-tool-template.jinja in the companion repo.

---

Load locally with Transformers

from transformers import AutoModelForCausalLM, AutoProcessor
import torch

model_id = "Swicked86/phi4-mm-gptq"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

Requires pip install llmcompressor (or pip install compressed-tensors) to load the quantization_config from the checkpoint.

---

Quality

Inference Tests

All tests run via vLLM on RTX 5090, --gpu-memory-utilization 0.45.

Text — factual recall (< 0.5s):

Prompt: "What is the capital of France?" Response: "The capital of France is Paris." ✅

Text — math reasoning (0.748s):

Prompt: "Solve step by step: If a train travels 120 miles in 2 hours, what is its speed in km/h?" Response: step-by-step solution → 96.54 km/h ✅

Text — code generation (1.068s):

Prompt: "Write a Python function that checks if a string is a palindrome." Response: correct is_palindrome() with docstring + example calls ✅

Vision — real image (3000×4000 JPEG):

Prompt: "Describe what you see in this image in detail." Response: correctly identified anime figure on a TV screen, described the room, entertainment setup, and animation style ✅

Audio — real voice message (Discord OGG Opus, converted to 16kHz WAV):

Input: Discord voice message (~11s) discussing software development Response: "The speaker is describing the process of transforming the rag function into a function that uses a local database rather than writing to and from files." ✅

Audio note: Discord voice messages are OGG Opus at 48 kHz. Convert to 16 kHz mono WAV before sending for best results. Pass "format": "wav" in the request.

vLLM LoRA note: vLLM currently only applies LoRA to the language model layers. Vision encoder LoRA layers (SigLIP) are silently skipped — this is a vLLM limitation. The speech LoRA (language decoder, rank-320) loaded and applied correctly.

---

Perplexity (wikitext-2-raw, context 512)

Benchmark will be added after upload.

---

Quantization Details

---

Architecture

---

Related

• Swicked86/phi4-mm-gguf — GGUF quantizations (text + vision, no audio)
• microsoft/Phi-4-multimodal-instruct — original bf16 model
• llm-compressor — quantization tool
• vLLM — inference engine