MiMo-V2.6-Distill-Qwen-9B EXL3 4.0 bpw

EXL3 4.0 bpw conversion of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B.

Build provenance

• Source revision: 2367e865d009c13ac81713a2878291d33ab28177
• ExLlamaV3: 1.5.2+cu128.torch2.10.0
• PyTorch: 2.10.0+cu128
• Target bitrate: 4.00 bpw
• Architecture: Qwen3_5ForConditionalGeneration
• Source license: MIT
• Artifact size: approximately 6.4 GiB

Validation

Validated locally on an NVIDIA GeForce RTX 4090.

The throughput figures are short local smoke measurements, not standardized cross-system benchmarks.

Tool-call validation produced a structured get_weather call for Toronto.

Vision validation used a generated test image containing a large red square; the model correctly returned red.

Source metadata compatibility notes

The source weights were not modified. Two local metadata normalizations were required for the current ExLlamaV3 conversion path:

1. The Qwen3.5 processor metadata declared Qwen2VLImageProcessor; the local conversion copy was normalized to Qwen2VLImageProcessorFast to match the ExLlamaV3 Qwen3.5 loader expectation.
2. The source config declared text_config.mtp_num_hidden_layers = 1, while the source safetensors index contained no MTP/NextN tensors. The local conversion copy therefore disabled the absent MTP side model.

These changes affect conversion metadata only and are recorded in the local build provenance.

TabbyAPI configuration used

model:
  backend: exllamav3
  max_seq_len: 8192
  cache_size: 8192
  gpu_split_auto: false
  gpu_split: [23, 0]
  vision: true
  reasoning: true
  reasoning_start_token: "<think>"
  reasoning_end_token: "</think>"
  tool_format: qwen3_5

The validation environment intentionally excluded the RTX 3060 from model placement so the reported 4090 measurements are not distorted by heterogeneous multi-GPU splitting.

Notes

This is an independent community quantization of the XiaomiMiMo upstream model. Refer to the upstream model card for intended use, limitations, and license terms.

RTX 4090 context scaling

TabbyAPI, RTX 4090 only, EXL3 4.0 bpw, FP16 cache, vision enabled, max batch size 1. Each context test used a unique long prompt containing a hidden needle near the end; the response had to begin with the exact needle code.

All six needle checks passed through about 64K prompt tokens. Decode throughput fell gradually from about 145 tok/s at 1K to 111 tok/s at 64K.

These are single-pass local smoke measurements, not standardized benchmark scores. Prefill throughput can move non-monotonically with kernel compilation, autotuning, cache state and prompt shape. The GPU-memory column is total observed GPU0 usage during the run, not an isolated model allocation.

Raw results: benchmarks/rtx4090-tabbyapi-context.json.

Long-generation / repetition stability

I also ran six long-generation samples through TabbyAPI:

• 3 runs with thinking disabled.
• 3 runs with thinking enabled and a 512-token reasoning budget.
• Up to 1,024 generated tokens per run.
• Observed decode throughput stayed around 145-146 tok/s.
• 0/6 runs triggered the repeated n-gram loop heuristic.
• The largest repeated 12-word-window count was 2.
• Repeated 4-gram ratios ranged from 0.0079 to 0.0381.
• 4/6 runs reached the 1,024-token ceiling, so this should not be interpreted as an EOS-behavior test.

Some outputs repeated short Markdown-formatting lines; those are retained in the raw data but are not counted as semantic repetition loops unless longer n-grams repeat as well.

This is a small stability smoke test, not proof that repetition loops cannot occur with other prompts or sampling settings.

Raw results: benchmarks/rtx4090-loop-stability.json.

Hardware comparison: RTX 4090 vs RTX 3060 vs mixed split

The same EXL3 artifact was tested three ways. These are local TabbyAPI smoke measurements, not standardized benchmark scores.

RTX 3060 12GB only

The model ran entirely on the RTX 3060 with no CPU model offload and no model layers placed on the RTX 4090. Vision was disabled for this accessibility test.

• 1K prompt: 53.5 tok/s, needle PASS
• 8K prompt: 49.0 tok/s, needle PASS
• ~16K prompt: 50.3 tok/s, needle PASS
• Short normal generation: 55.8 tok/s
• Repeated 4-gram ratio: 0.0
• Max repeated 12-gram count: 1
• Obvious repetition loop detected: no

Observed RTX 3060 memory was about 6.4-6.6 GiB during these runs. That is total observed GPU memory, not a model-only allocation measurement.

RTX 4090 + RTX 3060 mixed layer split

The mixed test used a conventional layer split with gpu_split [4, 8], not tensor parallelism.

• 1K prompt: 106.2 tok/s
• 8K prompt: 102.4 tok/s
• ~16K prompt: 100.7 tok/s
• Short normal generation: 108.1 tok/s
• Needle retrieval passed at all three context sizes.
• No obvious repetition loop was detected in the short generation smoke.

After load, total observed GPU memory was approximately 4.74 GiB on the RTX 4090 and 2.65 GiB on the RTX 3060.

For this 9B model, the mixed layer split is slower than a 4090-only placement because the model already fits comfortably on the faster card. The mixed result is still useful as a proof that heterogeneous layer splitting works and remains usable when a future model no longer fits on one GPU.

Raw results:

• benchmarks/rtx3060-tabbyapi.json
• benchmarks/rtx4090-rtx3060-mixed.json