Qwimi-3.6-27B-Coder-MTP-BF16

A coding-focused SFT of Qwen 3.6 27B, merged to full-precision BF16. Trained on a mixed corpus of coding, tool-calling, and agentic (SWE-agent) data in a single supervised fine-tuning run, with the coding domain deliberately kept dominant (~82.5% of training tokens) to avoid the smaller tool/agentic buckets degrading coding capability.

Status: public repository. This is the merged, ready-to-use model. For a smaller adapter-only download see Qwimi-3.6-27B-LoRA; for quantized GGUF files (llama.cpp / Ollama / LM Studio) see Qwimi-3.6-27B-GGUF.

---

Contents

1. Model summary
2. Intended use
3. Training configuration
4. Dataset composition
5. Training results
6. Chat template & tool-calling format
7. How to use
8. Verification & reproducibility
9. Known limitations
10. Benchmarks

---

1. Model summary

2. Intended use

Coding assistance (generation, debugging, refactoring, explanation) with visible chain-of-thought reasoning, plus native function/tool calling and basic agentic (SWE-agent style: multi-turn tool use over a repository) behavior.

Not validated for: image/vision input (untouched by this SFT — see §9), non-English use, safety-critical or high-stakes deployment, or any task outside code/tool/agentic reasoning.

3. Training configuration

4. Dataset composition

Single mixed SFT run — coding, tool-calling, and agentic examples trained together (not sequential stages), so the model learns all three without later stages overwriting earlier ones. Coding was deliberately kept dominant by token share, not just row count.

Coding (5 sources, deduplicated, content- or domain-filtered for code where the source had no reliable label; one low-quality synthetic source capped hard):

Tool-calling: minpeter/xlam-function-calling-60k-parsed — 6,000 rows (APIGen-generated, verifiable function calls, native tools schema). No chain-of-thought; direct tool calls.

Agentic (SWE-agent trajectories; the dominant length constraint in this run — most raw trajectories exceed 16,384 tokens and were dropped whole rather than truncated):

Final composition after dedup, 90/10 per-domain split, and whole-fit-or-drop length filtering at 16,384 tokens:

† One row was dropped by the trainer itself (all labels masked after truncation-safety filtering) — 22,359 rows actually trained on.

Reasoning-token fraction (coding + agentic, which carry <think> traces) was 95.2% of all training tokens, comfortably above the 75% floor generally recommended to preserve reasoning ability when mixing in non-reasoning (tool-calling) data.

5. Training results

Eval loss reported per domain via three held-out splits, evaluated every 150 steps (subsampled during training for speed: 400/1,778 coding rows, 200/595 tool rows, all 102/102 agentic rows — full-split numbers are pending a re-run and will be added if produced).

All three eval splits decreased monotonically across the entire run with no split turning up — no overfitting signal at 1 epoch. Coding, the dominant and highest-value domain, continued improving through the final checkpoint rather than being degraded by the smaller tool/agentic data — the outcome the mixed-run design targeted. Note that eval-loss magnitudes are not directly comparable across domains (different sequence structure and supervised-token counts per split); track each split against itself over time.

Single-batch training loss (0.500, 0.477, 0.493 at steps 750/900/1050) is noisy at effective batch 16 and is shown only as a secondary signal — eval loss is the trend to read.

6. Chat template & tool-calling format

Standard ChatML markers, with <think> reasoning canonically formatted by the tokenizer's chat template (verified — the template re-renders think content even if you pass it inline):

<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
<think>
{reasoning}
</think>

{answer}<|im_end|>

Tool calls use Qwen 3.6's native XML-style format (confirmed empirically against the installed chat template — this is not the JSON-object tool_call format used by earlier Qwen versions):

<|im_start|>assistant
<think>

</think>

<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call><|im_end|>

An empty <think></think> block is the canonical "direct response" form — tool calls in this model are trained to skip reasoning and act directly. Parallel tool calls render as sibling <tool_call> blocks. Pass your tool definitions via the standard tools= argument to apply_chat_template; the template injects a # Tools system block describing the exact call format automatically.

7. How to use

Recommended (tested path) — Unsloth:

from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained(
    "trjxter/Qwimi-3.6-27B-BF16",
    load_in_4bit=False,        # this repo is already full-precision BF16
    max_seq_length=16384,
)
tokenizer = processor.tokenizer
FastVisionModel.for_inference(model)

messages = [{"role": "user", "content": "Write a Python function to reverse a linked list."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, temperature=0.7, top_p=0.95)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Alternative — plain transformers (requires transformers>=5 for qwen3_5 support; confirm the exact model class name against your installed version's docs before relying on AutoModelForImageTextToText resolving correctly):

from transformers import AutoModelForImageTextToText, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("trjxter/Qwimi-3.6-27B-BF16")
model = AutoModelForImageTextToText.from_pretrained("trjxter/Qwimi-3.6-27B-BF16", dtype="bfloat16", device_map="auto")

This model was fine-tuned and validated on text-only input; no image inputs were used during training (see §9).

8. Verification & reproducibility

• Adapter integrity: final adapter adapter_model.safetensors = 1,867.8 MB, matching 466,911,232 trainable params × 4 bytes (fp32 save) exactly — confirms the saved file contains the complete, correctly-scoped trained adapter.
• Merge integrity: max |lora_B| (the check that the adapter actually trained before merging — an all-zero adapter would merge silently into an unchanged copy of the base) — PENDING, not yet recorded from the merge run. Treat the merge as provisionally verified pending this number.
• Tokenizer/config repair: the merged repo's tokenizer_config.json originally recorded a runtime-internal tokenizer_class value (TokenizersBackend, from transformers v5's tokenizer backend) that AutoTokenizer.from_pretrained cannot resolve. This was corrected in the repo to match the base model's tokenizer class — confirm your load succeeds before reporting issues; this repair should make plain AutoTokenizer loading work correctly.
• Checkpoint recovery: this run survived two Colab VM interruptions via checkpoints persisted to Drive every 150 steps; the final adapter reflects the complete 1,398-step run, not a recovered partial checkpoint.

9. Known limitations

• Vision tower untouched. The base checkpoint is multimodal; this SFT explicitly froze the vision layers (finetune_vision_layers=False) and trained on text-only data. No claims are made about image understanding — it is whatever the base model shipped with, unvalidated here.
• Agentic data is thin and short. After length-filtering, only 917 agentic training rows survived (12.7% of tokens), and DeepSWE trajectories longer than ~16k tokens were entirely excluded rather than truncated — this model has seen comparatively little of the longest, most complex agentic trajectories.
• One low-quality source, deliberately capped. Claude-opus-4.7-TraceInversion-5000x contains reasoning traces reverse-engineered by a small model from compressed summaries, not genuine teacher traces. Capped at 577 rows (3.5% of the coding split) specifically to limit this risk.
• 3.6% of coding examples were dropped for length (598 rows exceeding 16,384 tokens) — accepted as pathological outliers rather than truncated, since truncating a reasoning trace teaches the model to stop mid-thought.
• Context validated to 16,384 tokens despite the base model's much longer native context; behavior beyond that length is unvalidated by this training run.

10. Benchmarks

Qwimi was compared against the base Qwen 3.6 27B model using equivalent Q6 GGUF builds with MTP enabled and the same evaluation harness.

Two benchmark suites were run:

1. A fully auto-scored 300-task custom benchmark covering coding, tool-calling, and agentic workflows.
2. A broader 200-task mixed benchmark containing standard coding tasks and personal Qwimi-oriented evaluations.

Important: the latency figures below measure average end-to-end wall time per task, not raw tokens per second. SFT can reduce task completion time by producing shorter, more direct, better-formatted responses and reaching the stop condition sooner. The results do not prove that the underlying Q6 inference kernels became faster.

---

10.1 Custom 300-task benchmark

This benchmark contains 300 custom-authored, fully auto-scored tasks:

Both models answered the same 300 tasks, producing 600 total answer rows.

Overall results

The overall result is a small accuracy improvement combined with a very large reduction in task-completion time.

Category results

The strongest result was in custom coding, where Qwimi improved pass rate by four percentage points while completing tasks with 64.3% lower average wall time.

Tool-calling showed a smaller but positive accuracy gain. Agentic accuracy regressed by four points, although Qwimi still completed the agentic tasks substantially faster.

Pairwise outcomes

Among the 42 decisive tasks where only one model passed:

Category-level pairwise outcomes

300-task verdict: Qwimi achieved a slight overall quality win, driven by stronger coding and tool-calling performance, while also completing the suite in roughly half the time. The main remaining weakness was agentic-task accuracy.

---

10.2 Mixed 200-task benchmark

A separate 200-task evaluation compared the same Q6 MTP builds across standard coding tasks and personal Qwimi-oriented tests.

The run produced 400 raw answer rows, with 200 answers from each model. Some categories were omitted from the scored summary because their scorer/export integration was incomplete; they were not treated as zero-score failures.

Overall auto-scored results

Pairwise auto-scored results

On the 146 tasks with a valid paired auto-score:

Among the nine decisive paired tasks, Qwimi won:

[ \frac{7}{9} = 77.8% ]

This corresponds to a net pairwise edge of +3.42 percentage points across all paired auto-scored tasks.

Category highlights

200-task verdict: the result was directionally positive for Qwimi, particularly on HumanEval, XML-style tool behavior, and wall time. However, the overall pass-rate gain was small, most paired results were ties, and MBPP performance regressed.

---

10.3 Combined interpretation

Across both benchmark suites, Qwimi demonstrated:

• Higher overall pass rate in both runs.
• Stronger performance on custom coding, tool-calling, HumanEval, and personal XML-tool tasks.
• A positive pairwise record:
  • 23–19 on the 300-task custom benchmark.
  • 7–2 on decisive paired tasks in the 200-task benchmark.
• Substantially lower task-completion time:
  • 49.4% lower on the custom 300-task suite.
  • 23.2% lower on the mixed 200-task suite.
• A remaining weakness in some areas:
  • -4.00 points on custom agentic accuracy.
  • -3.85 points on MBPP.

The fairest overall conclusion is:

Qwimi is a modest but measurable coding/tool-use improvement over the base model, with unusually large task-level latency reductions. Its largest quality gains appear in coding and structured tool usage, while agentic and MBPP performance require further work.

---

10.4 Interpreting the speed results

Both evaluated models used Q6 quantization, so the speed difference should not be interpreted as Qwimi having fundamentally faster matrix-multiplication kernels.

Autoregressive task time can be approximated as:

[ T_{\text{task}} \approx T_{\text{prefill}} + N_{\text{generated tokens}} \times T_{\text{decode per token}} ]

Because both models use the same architecture family and quantization level, their raw decode cost per token should be broadly similar. SFT can still reduce total wall time by changing generation behavior:

• reaching the correct solution with fewer tokens;
• producing less unnecessary explanation or repetition;
• following the expected output format sooner;
• avoiding failed attempts, malformed tool calls, or self-correction loops;
• emitting the stop token earlier.

Therefore, the supported claim is:

Qwimi completed the benchmark tasks faster.

A separate controlled measurement of generated-token counts and tokens per second would be required to claim that Qwimi is intrinsically faster on a per-token basis.

---

10.5 Benchmark limitations

• Only the Q6 GGUF builds were compared; these results should not be assumed to apply identically to every quant listed in §2.
• Wall time depends on hardware, llama.cpp build, prompt length, output length, sampling settings, MTP support, and stop conditions.
• The 300-task suite is custom-authored and is not a replacement for established public benchmarks.
• Some categories in the 200-task run were excluded because their official scorer/export integration was incomplete; they were not counted as zero.
• Agentic tasks used keyword/structural auto-scoring and may not capture every aspect of repository-level software-engineering quality.
• More seeds, repeated runs, and token-level throughput measurements would improve confidence in the latency conclusions.