MiMo-V2.6-Distill-Qwen-9B — NVFP4 GGUF

NVFP4 quantization of Xiaomi's MiMo-V2.6-Distill-Qwen-9B, packaged for llama.cpp.

What this model is

MiMo-V2.6-Distill-Qwen-9B is a 9-billion-parameter agentic vision-language model built by Xiaomi MiMo. It is a supervised fine-tune (SFT) of Qwen3.5-9B on data generated by the larger MiMo-V2.6 model family, covering four target domains:

• Software engineering / agentic coding
• General-purpose agent work (tool use, terminal, automation)
• Visual coding (understanding and writing code from screen/UI images)
• Cybersecurity

Xiaomi released this SFT checkpoint as an open starting point for research in agentic reinforcement learning — the MiMo team's stated goal is to give the community a base checkpoint rather than a closed final product.

Model tree

Qwen/Qwen3.5-9B                       (base architecture, Alibaba Qwen)
└── XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B   (SFT by Xiaomi MiMo, MIT)
    └── FreedomAISVR/MiMo-V2.6-Distill-Qwen-9B-NVFP4-GGUF   (this repo)

Source: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B · License: MIT · Architecture: Qwen3_5ForConditionalGeneration

Architecture

• Hybrid Mamba-2 + attention: 32 layers — 24 linear-attention (Mamba-2) layers and 8 full-attention layers (full_attention_interval: 4). The Mamba layers are why 256K context is affordable in VRAM.
• Vision encoder included (333 vision tensors, patch 16, depth 27) — image and video inputs, shipped separately as mmproj.
• Context: 262,144 tokens (256K) — max_position_embeddings: 262144, inherited from Qwen3.5-9B.
• Hidden 4096, 16 heads / 4 KV heads, head_dim 256, intermediate 12288, vocab 248,320, rope θ 10M with 0.25 partial rotary, interleaved MRoPE.
• Thinking is enabled by default (source chat template) — responses begin with [Start thinking].

Reported results (from Xiaomi's model card)

Trained on a 77.4B-token SFT mixture (27.2B loss-bearing) split across code, cyber, general, and visual data. Figures above are Xiaomi's own reported numbers, not reproduced here.

Quantization

• BPW is measured against the F16 GGUF (16.69 GiB, 427 tensors), not an idealized FP32 baseline.
• Norms, embeddings, and per-layer scales stay F32 by design — NVFP4 stores weights only.
• NVFP4 = GGUF type 40 / llama.cpp ftype 39. Hardware acceleration requires NVIDIA Blackwell (RTX 50 series); older GPUs fall back to slower paths.

Inference performance

Measured on NVIDIA GeForce RTX 5060 Ti 16 GB (Blackwell SM120), CUDA 13.2, llama.cpp build b1868-4fea119d, full GPU offload (-ngl 99), KV cache Q8_0 for both K and V.

Notes on these numbers:

• The 128K prompt figure is from a ~20-token prompt, so it reflects a tiny batch and is not representative of long-prompt throughput.
• Prompt throughput at 256K reflects large-batch processing of the full 258K-token input.
• Generation speed drops at 256K because the KV cache is roughly twice as large — expected, not a defect.
• At 256K the model followed instructions correctly and stayed coherent across the full context.

Load test: ✅ loads and generates · Vision test: ✅ llama-mtmd-cli + mmproj correctly described a test image · Chat template: ✅ embedded (tokenizer.chat_template)

Usage

llama.cpp CLI (text):

llama-cli -m mimo-v2.6-distill-qwen-9b-nvfp4.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99

llama.cpp CLI (vision):

llama-mtmd-cli -m mimo-v2.6-distill-qwen-9b-nvfp4.gguf \
  --mmproj mmproj-mimo-v2.6-distill-qwen-9b-f16.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 \
  --image your_image.png

llama.cpp server:

llama-server -m mimo-v2.6-distill-qwen-9b-nvfp4.gguf \
  -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 --port 8080

llama-cpp-python: requires a wheel built against a llama.cpp with NVFP4 support — stock PyPI wheels will not load this file. No quantization-aware Python package is published for it yet.

For grounding tasks with images, llama.cpp suggests --image-min-tokens 1024 (upstream issue #16842).

Pipeline

# 1. F16 conversion (--no-mtp is required, see below)
python convert_hf_to_gguf.py <source_dir> --outfile model-f16.gguf --outtype f16 --no-mtp
# 2. vision encoder
python convert_hf_to_gguf.py <source_dir> --outfile mmproj-model-f16.gguf --outtype f16 --mmproj
# 3. quantize
llama-quantize model-f16.gguf model-nvfp4.gguf NVFP4

Model tree / provenance

Qwen/Qwen3.5-9B → XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B → this repo (quantization only).

Quantization does not change the model's weights, knowledge, or training — it only re-encodes them at lower precision. It does not improve the model.

Known limitations

• No MTP head. The source config declares mtp_num_hidden_layers: 1, but the released checkpoint contains no MTP weights, so this GGUF is converted with --no-mtp (block_count: 32). Multi-token prediction is unavailable; this is a property of the released checkpoint, not a conversion loss.
• Thinking is on by default — the upstream template emits [Start thinking] unless the caller disables it.
• This is an SFT research checkpoint, not a polished production model; Xiaomi positions it as a starting point for agentic RL.
• Benchmark numbers are Xiaomi's reported results and were not reproduced during quantization.
• KV cache at 256K is large — on 16 GB cards, Q8 KV leaves limited headroom for long generations.
• NVFP4 requires Blackwell for hardware-accelerated inference.