GEHIRNS/magi-1.0

Paper: Where the Bits Go (Sun, 2026) · Recipe and data: GEHIRNS/magi · Website: gehirns.pages.dev

Introduction

magi-1.0 is Qwen3.5-4B in 1.80 GiB. Served with an 8-bit KV cache, it keeps 96% of the full model's benchmark score in 30% of the GPU memory (97% in 34% with the default cache) and decodes 2.7× faster, with no training: the bits are simply placed where the model needs them.

magi-1.0 stores Qwen/Qwen3.5-4B in a mixture of standard GGUF quantization types. Each tensor class gets the precision that its measured sensitivity calls for: more bits for the small attention and DeltaNet projections that break easily, fewer for the large FFN matrices that tolerate them. The weights are Qwen's own. The file averages 3.65 bits per weight and runs unmodified in llama.cpp and Ollama.

Capability density

Capability density is ρ = −log₂(1 − score) / GiB: how many times the benchmark error rate is halved per gibibyte of GPU memory. Memory is the resident size that Ollama's /api/ps reports with a 32K context loaded (weights, KV cache, recurrent state and compute buffers, without the CUDA context, which is about 1.05 GiB for every model). magi-1.0 + q8 KV is magi-1.0 served with an 8-bit KV cache. The 95% intervals, from a suite-stratified bootstrap over tasks, are 0.856–0.934 for magi-1.0 + q8 KV, 0.768–0.835 for magi-1.0, 0.673–0.734 for Q4_K_M and 0.646–0.703 for Q2_K.

Unsloth's Q4_K_M matches BF16's accuracy (84.14%) and is 2.85 points more accurate than magi-1.0. It needs 0.76 GiB more memory, or 1.13 GiB more than magi-1.0 + q8 KV. If accuracy matters most, use Q4_K_M. If memory is the constraint, magi-1.0 gives you 14–27% more capability per GiB. Against the published Q2_K file, magi-1.0 is smaller and 4.2 points more accurate (paired 95% interval 2.8–5.6).

Benchmarks

All models were run on one RTX 5070 (12 GB) under the same protocol: thinking on with a 4,096-token budget (budget forcing), greedy decoding, 32K context, 2,313 tasks. Code is graded by executing tests (EvalPlus for HumanEval+ and MBPP+); LiveCodeBench uses 150 problems from contests between 2025-01-18 and 2025-04-06. Small numbers in the Ours columns are relative to BF16.

	Baselines			Ours
Benchmark	BF16 _{8.98 GiB}	Q4_K_M _{3.78 GiB}	Q2_K _{3.15 GiB}	magi-1.0 _{3.02 GiB}	magi-1.0 + q8 KV _{2.65 GiB}
Efficiency
GPU memory at 32K (GiB)	8.98	3.78	3.15	3.02_{3.0× less}	2.65_{3.4× less}
Benchmark score	83.96%	84.14%	77.06%	81.30%_97%	80.64%_96%
Capability density	0.294	0.703	0.675	0.800_2.7×	0.893_3.0×
Decode speed (tok/s)	66.3	146.6	165.5	180.0_2.7×	181.4_2.7×
Traces ending in a loop	3.4%	3.3%	33.3%	9.4%	9.1%
Coding
HumanEval+	91.46	92.68	75.00	87.80	88.41
MBPP+	72.18	73.68	58.90	68.42	67.92
LiveCodeBench	35.33	35.33	15.33	24.67	24.00
Math
GSM8K	87.33	93.33	90.67	89.33	87.33
Science and Knowledge
ARC-Challenge	98.67	98.67	97.33	97.33	98.00
MMLU	84.50	82.50	74.00	82.00	81.00
OpenBookQA	95.00	96.50	93.00	96.00	95.00
SciQ	98.00	98.50	96.00	96.00	97.00
TruthfulQA	82.50	82.00	73.00	79.50	76.00
Commonsense
HellaSwag	75.50	72.00	71.50	72.00	69.00
WinoGrande	87.00	84.50	80.00	82.50	84.00
Tool Use
Function calling	100.00	100.00	100.00	100.00	100.00

The largest gap to BF16 is competitive programming (LiveCodeBench). Most of magi-1.0's advantage over Q2_K comes from looping less: 337 of Q2_K's 575 failed tasks end with the reasoning stuck in a repetition loop, against 99 of magi-1.0's 463. Over the whole benchmark, a task takes magi-1.0 13.8 seconds on average, against 16.5 for Q4_K_M and 36.1 for BF16.

Recipe

Tensor group	Type	Share of parameters	bits/weight
FFN gate, up, down (all 32 layers)	IQ3_XXS	53.9%	3.06
Gated-attention output gate (24 DeltaNet layers)	IQ3_XXS	6.0%	3.06
DeltaNet input projection `attn_qkv` (24)	Q4_K	12.0%	4.50
DeltaNet output projection `ssm_out` (24)	Q4_K	6.0%	4.50
Full-attention q, k, v, output (8 layers)	Q4_K	7.0%	4.50
Token embedding = tied output head	Q4_K	15.1%	4.50
DeltaNet gates `ssm_alpha`, `ssm_beta`	Q8_0	0.1%	8.50
Norms, convolution, A, Δt bias	F32	<0.1%	32

Types were allocated greedily under a 1.80 GiB budget by the measured increase in greedy next-token disagreement with BF16 per gibibyte, scored on the model's own long reasoning traces and chat answers. The importance matrix comes from 32 of the model's own reasoning traces. At the same file size, this allocation disagrees with BF16 on 10.0% of tokens (10.2% with the importance matrix the random allocations used), against 14.3–17.5% for three random allocations of the same bytes.

How to run

llama.cpp (uses the Qwen3.5 chat template embedded in the file):

llama-server -m magi-1.0.gguf -c 32768 --jinja
# 8-bit KV cache (magi-1.0 + q8 KV, 2.65 GiB):
llama-server -m magi-1.0.gguf -c 32768 --jinja --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0

Ollama (verified with Ollama 0.40.3, which picks up the embedded template):

ollama create magi-1.0 -f Modelfile
ollama run magi-1.0

For the 8-bit KV cache, start the server with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0. The benchmark used greedy decoding. For interactive use with thinking enabled, Qwen recommends temperature 0.6, top_p 0.95 and top_k 20.

Limitations

Evaluated on one model family, one runtime (Ollama), one benchmark and one GPU; each configuration was run once.
Below BF16 in accuracy, most clearly on LiveCodeBench (24.7% vs 35.3%). 9.4% of thinking traces end in a repetition loop, against BF16's 3.4%.
Text only: the vision encoder of the base model is not included.
The name is ours. magi-1.0 is not a fine-tune and contains no new training.

GEHIRNS/magi-beta-1: an alternative allocation with the full-attention k, v and output projections at Q6_K (80.65%, capability density 0.771), published for comparison.
Paper: Eason Sun, Where the Bits Go: Sensitivity-Allocated Mixed Precision for a 4B Reasoning Model on a 12 GB Consumer GPU, Zenodo, 2026. DOI: 10.5281/zenodo.23630199.
GEHIRNS/magi on GitHub: the build recipe, the importance matrix and the data behind every table and figure of the paper.
gehirns.pages.dev: the GEHIRNS website.

Citation

@misc{sun2026wherethebitsgo,
  title     = {Where the Bits Go: Sensitivity-Allocated Mixed Precision for a 4B Reasoning Model on a 12 GB Consumer GPU},
  author    = {Sun, Eason},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23630199},
  url       = {https://doi.org/10.5281/zenodo.23630199}
}

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B. The license text is included in this repository.