Qwimi-3.6-27B-Coder-MTP-GGUF

GGUF quantizations of Qwimi-3.6-27B-Coder-MTP-BF16 — a coding-focused SFT of Qwen 3.6 27B (mixed coding + tool-calling + agentic training; see Qwimi-3.6-27B-BF16 for the full model card, training configuration, dataset composition, and eval results). This repo exists for llama.cpp and downstream tools (Ollama, LM Studio, text-generation-webui, etc.).

Status: public repository. All 10 planned quants are uploaded and confirmed.

---

Contents

1. Source & conversion
2. Available quants
3. Which quant should I use?
4. Important: llama.cpp version requirement
5. How to run
6. Tool-calling with llama.cpp
7. Training summary
8. Known limitations
9. Benchmarks

---

1. Source & conversion

• Source: trjxter/Qwimi-3.6-27B-BF16 (merged full-precision model)
• Converted with: llama.cpp — convert_hf_to_gguf.py --outtype bf16, then llama-quantize per target quant
• Build: CPU-only (GGML_CUDA=OFF); quantization is a CPU-bound process and does not require GPU
• Conversion note: the merged repo's tokenizer_config.json originally carried a transformers-v5-internal tokenizer_class value that AutoTokenizer (used internally by the converter) could not resolve. This was corrected before conversion; if you reproduce this conversion yourself from an older snapshot of the BF16 repo, verify tokenizer_config.json reflects the base model's tokenizer class first.

2. Available quants

All sizes confirmed from the uploaded files (Hugging Face Hub hardware-compatibility widget, computed from actual repo blobs). Total repo size: 175.0 GB across 10 quants.

Q2_K_L naming note: this is not a standard llama-quantize type — it's the community recipe of a Q2_K base with the token-embedding and output (LM head) tensors held at Q8_0 precision (--token-embedding-type Q8_0 --output-tensor-type Q8_0), which meaningfully improves quality over plain Q2_K at a small size cost. All other quants above are standard llama.cpp K-quant types.

Not produced: Unsloth "UD" (Dynamic 2.0) quants and sub-3-bit i-quants (IQ2_*, IQ3_XXS) are not included — these require Unsloth's proprietary calibrated quantization pipeline, not plain llama-quantize, and are out of scope for this repo.

3. Which quant should I use?

General guidance for this model family and size (27.8B):

• Best balance of quality/size: Q4_K_M — the standard recommended default for most use.
• Near-lossless, if you have the VRAM/RAM: Q8_0 (~30 GB) or Q6_K (~23 GB).
• Constrained VRAM: Q4_K_S or Q3_K_M/Q3_K_L — expect a noticeable but usually acceptable quality drop versus Q4_K_M.
• Extreme constraint: Q2_K_L — usable, but at 2-bit the model's coding accuracy and instruction-following will degrade more than the size savings suggest; only recommended if no larger quant fits your hardware.

This model was trained and merged in BF16; no quant-specific fine-tuning was performed, so relative quality ordering follows standard llama.cpp quantization behavior for models this size — it has not been separately benchmarked per quant here (see §9).

4. Important: llama.cpp version requirement

Qwen3.6-27B (model_type: qwen3_5) is a hybrid architecture — standard attention layers interleaved with GatedDeltaNet linear-attention layers. Support for this architecture in llama.cpp and downstream tools (Ollama, LM Studio, text-generation-webui) depends on how recently they've pulled in qwen3_5/GatedDeltaNet support upstream. Verify your llama.cpp build (or wrapping tool) explicitly supports this architecture before assuming compatibility — an outdated build may fail to load the model or silently misinterpret the linear-attention layers. If you hit load errors, check for a llama.cpp update first.

5. How to run

# llama.cpp CLI
./llama-cli -m Qwimi-3.6-27B-Q4_K_M.gguf -p "Write a Python function to reverse a linked list." -n 512

# llama-server (OpenAI-compatible API)
./llama-server -m Qwimi-3.6-27B-Q4_K_M.gguf --ctx-size 16384

Set --ctx-size to at most 16384 — that is the sequence length this model was trained and validated at, even though the base architecture supports a much longer native context. Behavior beyond 16,384 tokens is unvalidated.

6. Tool-calling with llama.cpp

This model uses Qwen 3.6's native XML-style tool-call format (<tool_call><function=...> <parameter=...>), not the JSON-object format from earlier Qwen versions — see the full example in the Qwimi-3.6-27B-BF16 README §6. For llama-server to apply the model's chat template (including tool definitions) correctly, you need a recent build with Jinja chat template support enabled (commonly a --jinja flag) — verify against your installed llama.cpp version's documentation, since flag names and defaults change across releases.

7. Training summary

Condensed from the full model card (Qwimi-3.6-27B-BF16) — see that repo for complete details, dataset composition, and eval curves.

• Base: unsloth/Qwen3.6-27B, dense 27.8B, text-only SFT (vision frozen)
• Method: 4-bit QLoRA, r=64/alpha=64, language layers only
• Data: mixed single-run SFT — coding (82.5% of tokens) + tool-calling (4.8%) + agentic SWE-agent trajectories (12.7%), 22,359 total training rows, 72.5M training tokens
• 1 epoch, effective batch 16, 1,398 optimizer steps, A100 80GB
• Final eval loss: coding 0.4761, tool 0.0208, agentic 0.2892 — all three domains improved monotonically through the full run with no overfitting signal

8. Known limitations

See Qwimi-3.6-27B-BF16 §9 for the full list (vision untouched, thin agentic coverage, one capped low-quality coding source, 3.6% of coding examples dropped for length rather than truncated, 16,384-token validated context). Additionally for this GGUF repo specifically:

• Quantization introduces its own quality loss on top of the base model's characteristics, increasing with lower bit-width — see §3 for guidance.
• This is a fresh/bleeding-edge architecture in llama.cpp (§4) — expect rougher edges in tooling support than for long-established architectures.

9. Benchmarks

Qwimi was compared against the base Qwen 3.6 27B model using equivalent Q6 GGUF builds with MTP enabled and the same evaluation harness.

Two benchmark suites were run:

1. A fully auto-scored 300-task custom benchmark covering coding, tool-calling, and agentic workflows.
2. A broader 200-task mixed benchmark containing standard coding tasks and personal Qwimi-oriented evaluations.

Important: the latency figures below measure average end-to-end wall time per task, not raw tokens per second. SFT can reduce task completion time by producing shorter, more direct, better-formatted responses and reaching the stop condition sooner. The results do not prove that the underlying Q6 inference kernels became faster.

---

9.1 Custom 300-task benchmark

This benchmark contains 300 custom-authored, fully auto-scored tasks:

Both models answered the same 300 tasks, producing 600 total answer rows.

Overall results

The overall result is a small accuracy improvement combined with a very large reduction in task-completion time.

Category results

The strongest result was in custom coding, where Qwimi improved pass rate by four percentage points while completing tasks with 64.3% lower average wall time.

Tool-calling showed a smaller but positive accuracy gain. Agentic accuracy regressed by four points, although Qwimi still completed the agentic tasks substantially faster.

Pairwise outcomes

Among the 42 decisive tasks where only one model passed:

Category-level pairwise outcomes

300-task verdict: Qwimi achieved a slight overall quality win, driven by stronger coding and tool-calling performance, while also completing the suite in roughly half the time. The main remaining weakness was agentic-task accuracy.

---

9.2 Mixed 200-task benchmark

A separate 200-task evaluation compared the same Q6 MTP builds across standard coding tasks and personal Qwimi-oriented tests.

The run produced 400 raw answer rows, with 200 answers from each model. Some categories were omitted from the scored summary because their scorer/export integration was incomplete; they were not treated as zero-score failures.

Overall auto-scored results

Pairwise auto-scored results

On the 146 tasks with a valid paired auto-score:

Among the nine decisive paired tasks, Qwimi won:

[ \frac{7}{9} = 77.8% ]

This corresponds to a net pairwise edge of +3.42 percentage points across all paired auto-scored tasks.

Category highlights

200-task verdict: the result was directionally positive for Qwimi, particularly on HumanEval, XML-style tool behavior, and wall time. However, the overall pass-rate gain was small, most paired results were ties, and MBPP performance regressed.

---

9.3 Combined interpretation

Across both benchmark suites, Qwimi demonstrated:

• Higher overall pass rate in both runs.
• Stronger performance on custom coding, tool-calling, HumanEval, and personal XML-tool tasks.
• A positive pairwise record:
  • 23–19 on the 300-task custom benchmark.
  • 7–2 on decisive paired tasks in the 200-task benchmark.
• Substantially lower task-completion time:
  • 49.4% lower on the custom 300-task suite.
  • 23.2% lower on the mixed 200-task suite.
• A remaining weakness in some areas:
  • -4.00 points on custom agentic accuracy.
  • -3.85 points on MBPP.

The fairest overall conclusion is:

Qwimi is a modest but measurable coding/tool-use improvement over the base model, with unusually large task-level latency reductions. Its largest quality gains appear in coding and structured tool usage, while agentic and MBPP performance require further work.

---

9.4 Interpreting the speed results

Both evaluated models used Q6 quantization, so the speed difference should not be interpreted as Qwimi having fundamentally faster matrix-multiplication kernels.

Autoregressive task time can be approximated as:

[ T_{\text{task}} \approx T_{\text{prefill}} + N_{\text{generated tokens}} \times T_{\text{decode per token}} ]

Because both models use the same architecture family and quantization level, their raw decode cost per token should be broadly similar. SFT can still reduce total wall time by changing generation behavior:

• reaching the correct solution with fewer tokens;
• producing less unnecessary explanation or repetition;
• following the expected output format sooner;
• avoiding failed attempts, malformed tool calls, or self-correction loops;
• emitting the stop token earlier.

Therefore, the supported claim is:

Qwimi completed the benchmark tasks faster.

A separate controlled measurement of generated-token counts and tokens per second would be required to claim that Qwimi is intrinsically faster on a per-token basis.

---

9.5 Benchmark limitations

• Only the Q6 GGUF builds were compared; these results should not be assumed to apply identically to every quant listed in §2.
• Wall time depends on hardware, llama.cpp build, prompt length, output length, sampling settings, MTP support, and stop conditions.
• The 300-task suite is custom-authored and is not a replacement for established public benchmarks.
• Some categories in the 200-task run were excluded because their official scorer/export integration was incomplete; they were not counted as zero.
• Agentic tasks used keyword/structural auto-scoring and may not capture every aspect of repository-level software-engineering quality.
• More seeds, repeated runs, and token-level throughput measurements would improve confidence in the latency conclusions.