Gwimi-4-12B-IT-GGUF

Quantized GGUF releases of Gwimi-4-12B-IT, a Gemma 4 12B instruction model post-trained through:

1. Supervised Fine-Tuning (SFT) on a 20,000-example reasoning mixture.
2. Group Sequence Policy Optimization (GSPO) on 12,000 frozen reinforcement-learning prompts.

The source model for every file in this repository is:

trjxter/Gwimi-4-12B-IT-BF16

The BF16 release contains the cumulative SFT + GSPO updates merged into the exact original Gemma 4 12B BF16 base. No LoRA adapter is required when using these GGUF files.

---

Available quantizations

Quick recommendation

• Best general default: Q4_K_M
• Better quality with moderate extra memory: Q5_K_M
• High-quality local inference: Q6_K
• Closest to the BF16 model: Q8_0
• Memory-constrained systems: Q3_K_M or Q3_K_L
• Absolute smallest file: Q2_K_L

File size is not the same as total runtime memory usage. Your runtime also needs memory for the inference backend, model metadata, temporary buffers, and the KV cache. Longer context lengths increase KV-cache memory use.

---

Model training overview

Stage 1: Supervised Fine-Tuning

The SFT corpus contained exactly 20,000 examples.

SFT dataset composition

The Kimi K2.5 top-up consisted of:

The final split contained:

SFT configuration

SFT evaluation

The 2,000-example evaluation set contained:

• 100 fixed anchor examples
• 1,900 examples in a rotating evaluation pool

Scheduled evaluations used 100 fixed anchor examples plus 100 rotating examples.

The final full held-out evaluation produced:

These are teacher-forced SFT evaluation metrics and should not be interpreted as free-generation benchmark accuracy.

---

Stage 2: GSPO reinforcement learning

GSPO stands for Group Sequence Policy Optimization.

For each prompt, the model generated multiple candidate completions. Programmatic reward functions scored those completions, and training used within-group reward differences to update the policy.

The defining configuration was:

importance_sampling_level = "sequence"

This applies sequence-level rather than token-level importance ratios.

GSPO datasets

Training split

Reserved evaluation and protection splits

The frozen dataset suite had zero cross-split normalized-prompt overlap.

Reward functions

The run used three frozen reward components:

correctness_reward_func
format_reward_func
anomaly_reward_func

They measured:

• answer correctness;
• required response formatting;
• malformed, repetitive, degenerate, or suspicious generations.

GSPO configuration

Near-terminal GSPO telemetry

The following values are a near-terminal W&B snapshot around global step 2250. They are training telemetry, not independent benchmark results.

The run stopped at global step 2250 after the format reward had saturated, completion lengths remained controlled, entropy had stabilized, and sequence clipping remained active without saturating.

The online RL worker did not run periodic held-out evaluation during the expensive generation loop. The preserved anchor, held-out, and protected test splits are intended for later independent evaluation.

---

Quantization provenance

All GGUF files were generated from the same verified merged BF16 model:

trjxter/Gwimi-4-12B-IT-BF16

The merged BF16 weights were verified to differ from the untouched base model:

Base SHA-256:
5a84cb313260ac447237b890387116dfa8682e49a6b44bc585ae8353abbff18d

Merged SHA-256:
1e024792bf994c200fc7757621d202eb2bb2ba11593afcf6a6a98ab6bb9c4845

The original and merged BF16 files had the same byte size because LoRA merging changes the numerical values inside the existing model tensors rather than adding a second set of full model weights.

A temporary BF16 GGUF was used as the private quantization source. It was not uploaded to this repository.

All 12 public GGUF files were generated and then verified as present in this repository.

---

Running with llama.cpp

Use a recent llama.cpp build with Gemma 4 GGUF support.

Example using Q4_K_M:

llama-cli \
  -m Gwimi-4-12B-IT-Q4_K_M.gguf \
  -cnv \
  -c 4096 \
  -n 1024 \
  --temp 0.7 \
  --top-p 0.95

For a direct prompt:

llama-cli \
  -m Gwimi-4-12B-IT-Q4_K_M.gguf \
  -p "Solve carefully: What is 17% of 240?" \
  -c 4096 \
  -n 512 \
  --temp 0.7 \
  --top-p 0.95

Adjust GPU offloading according to your hardware. For example:

-ngl 99

attempts to offload as many model layers as possible to the GPU.

---

Using the model in local applications

These files are intended for GGUF-compatible runtimes such as:

• llama.cpp
• LM Studio
• KoboldCpp
• compatible local model launchers and servers

Select the quant that fits comfortably within your available RAM or VRAM after accounting for KV cache and runtime overhead.

For most users, begin with:

Gwimi-4-12B-IT-Q4_K_M.gguf

Then compare Q5_K_M, Q6_K, or Q8_0 when more memory is available.

---

Intended uses

This model is intended for experimentation with:

• mathematical and scientific reasoning;
• coding and debugging assistance;
• technical question answering;
• structured instruction following;
• local inference;
• quantization-quality comparisons;
• SFT and reinforcement-learning research.

---

Evaluations

Independent free-generation benchmarks were run locally against Gwimi-4-12B-IT-Q6_K.gguf using llama.cpp. Results below are pass@1 with greedy decoding unless noted otherwise.

Run date: 2026-06-20

Common settings

Summary

HumanEval is the only benchmark completed end-to-end. The other four runs are partial subsets; totals in the right-hand column are the intended full benchmark size for that eval configuration.

LiveCodeBench v6 — by difficulty

MultiPL-E — by language

MMLU-Pro — by category

Fifty questions were drawn from a stratified 2,000-question pool (12,032 total in the test split). Three answers failed letter extraction and are counted as incorrect.

These results reflect one quantization (Q6_K) under one local inference setup. They should not be compared directly to leaderboard numbers from other backends, quantizations, or sampling settings without matching those conditions.

PersonalBenchmarks (custom suite)

A 56-scenario local suite covering reasoning, tool calling, agentic planning, structured output, long-context recall, programming, and response-efficiency constraints. Scoring is deterministic (JSON shape checks, keyword order, executable patterns) — not LLM-as-judge.

Suite version: v2 (2026-06-21) — Performance, Tool Calling, and Agentic use canonical prompts with explicit raw-JSON format instructions (same scenarios across all model comparisons). Prior v1 prompts are archived under PersonalBenchmarks/scenarios/archive/.

Run date: 2026-06-21 · Settings: Q6_K, llama.cpp @ http://127.0.0.1:8088/v1, temperature 0.0, thinking on, seed 42, n_predict 8192

Performance, Tool Calling, and Agentic scores are from the v2 canonical scenario run. Reasoning, Structured Output, Context, and Programming scores are from the prior 2026-06-20 run (scenarios unchanged). Re-run python run_personal.py for a single fresh 56-scenario pass on any model.

PersonalBenchmarks is maintained in this repository (PersonalBenchmarks/, run_personal.py). It is independent of the third-party BenchLocal desktop benchmark packs.

---

Limitations

1. External benchmarking is still in progress: HumanEval is complete, but MBPP, LiveCodeBench, MultiPL-E, and MMLU-Pro are partial runs on Q6_K only.
2. GSPO reward telemetry is not equivalent to external benchmark accuracy.
3. Reward optimization can inherit blind spots from the reward functions.
4. The RL stage focused heavily on verifiable mathematics, science, and structured reasoning.
5. Long-context behavior was not independently benchmarked after GSPO.
6. Quantization can change generation quality, especially at lower bit widths.
7. The model may hallucinate, make calculation errors, produce unsafe advice, or follow incorrect premises.
8. Outputs should be independently verified for high-stakes use.
9. Results can vary across llama.cpp versions, inference backends, hardware, context lengths, and sampling settings.

---

Recommended evaluation approach

For a fair comparison, evaluate these models under identical prompts and decoding settings:

1. Original Gemma 4 12B instruction base
2. Gwimi SFT-only checkpoint
3. Gwimi SFT + GSPO BF16 model
4. Each selected GGUF quantization

Keep fixed:

• prompt formatting;
• chat template;
• maximum generated tokens;
• temperature and top-p;
• random seed;
• answer extraction;
• benchmark scoring;
• context length;
• inference backend where practical.

Useful comparisons include:

• exact-answer mathematics;
• scientific multiple choice;
• coding and debugging;
• formatting compliance;
• repetition and anomaly rate;
• response length;
• pass@1 and sampled pass@k;
• generation speed;
• RAM and VRAM use;
• qualitative reasoning review.

---

Acknowledgements

This release builds on:

• Gemma;
• Unsloth;
• Hugging Face;
• Transformers, PEFT, and TRL;
• llama.cpp;
• Math-Verify;
• the authors and maintainers of the SFT and GSPO datasets.

---

License

This model is a derivative of Gemma and remains subject to the applicable Gemma license and terms of use.

Users are responsible for reviewing the upstream license and ensuring that their intended use complies with it.