small-test: a 95M multimodal fixture for the llama.cpp server CI

Idea and first version: Xuan-Son Nguyen (ngxson). Recovery training, data pipeline, evaluation and this release: Pascal (Serveurperso). Built together, one machine each, over a week of evenings.

One model that exercises every path of llama-server in one go: chat, tool calling in the Qwen3.5 XML format, OCR through mmproj, and speculative decoding with its own MTP head as the draft model. 82M text parameters, 12.6M MTP head, 58M vision tower. 190 MB in f16, 102 MB in Q8_0. It is deterministic in greedy mode: the 19 server tests below pass identically run after run.

This is a test fixture, not an assistant. It is a pruned Qwen3.5-0.8B with 6 of 31 layers, trained to be predictable on the shapes the llama.cpp CI asks for. It will happily tell you that C++ is simple and easy to use.

Files

Use in llama.cpp

llama-server -hf Serveurperso/small-test --mmproj-url https://huggingface.co/Serveurperso/small-test/resolve/main/small-test-mmproj-f16.gguf --jinja --spec-type draft-mtp

In tools/server/tests the model is ServerPreset.small_test() and unit/test_small_test.py runs 19 tests against it: required and auto tool calls, tool results, no tool call cases, OCR on a rendered image, and MTP draft equivalence with the target. About 12 seconds on CPU.

Architecture

Qwen3.5-0.8B pruned to layers [0, 11, 14, 19, 22, 23] of 31, chosen by leave one out loss on a text subset, alternating linear (Gated DeltaNet) and full attention. FFN cut to 1024. Vocabulary pruned from 150k to 24935 tokens by byte level BPE frequency with merge closure (min_count 600). The MTP head is trained jointly at weight 0.3 and used as the draft model (--spec-type draft-mtp). The vision tower keeps 6 blocks and is trained jointly on synthetic OCR images. Nothing is randomly initialized: every surviving weight comes from the parent, and the loss starts at ln(vocab) = 10.1 because the pruned chain has to be reconnected.

Two constraints found the hard way: llama.cpp enables thinking by default on Qwen3.5 templates, so the template ships with all reasoning handling removed and tokenizer_config.json carries no chat_template key (gguf-py prefers it over the jinja file). And mtmd letterboxes images with black padding where the HF processor stretches them, so OCR training uses the llama.cpp preprocessing for 75 percent of samples.

Training

One script (train.py), one cosine schedule of 17000 steps at batch 32 x 2048, warmup 200, Muon at 2e-3 on the 2D hidden matrices and AdamW at 1e-3 on embeddings and norms, weight decay 0.01, fp32 master weights with bf16 autocast. Steps 0 to 6000 are text only, steps 6000 to 17000 add synthetic OCR images on 35 percent of packed documents. Fused Liger cross entropy on assistant spans only, documents packed whole and never truncated. 1.1B tokens, 2 hours on one RTX PRO 6000 at 150k tokens per second in the text phase.

Data mix in tokens: general chat from smol-smoltalk 38 percent, single turn tool calling from apigen 7 percent, Hermes function calling 3 percent capped at a few epochs, nvidia/Nemotron-Agentic-v1 tool calling trajectories under 2048 tokens with reasoning stripped 27 percent, a procedural generator of multi step tool trajectories 23 percent (synth_tools.py), and 1.5 percent of documents shaped like the CI scenarios themselves (ci_shapes.py, built from the test definitions).

The generator produces random tool schemas with distractor examples in the descriptions, chains of 2 to 7 steps where every argument is copied either from the user prompt or from a named field of an earlier tool result, parallel calls, threshold gates, conditional last steps that are skipped half the time, follow up turns, vague prompts with a single tool, a code execution tool with real Python, weather lookups with the location copied verbatim, questions about the tools answered in plain text, and random system prompts. Entity and tool names never overlap the CI tests.

What we measured and what failed

The starting point scored 4/10 on scripts/server-test-function-call.py. Its training data had no document with two or more tool calling turns at all: the Hermes "multi turn" set was multi turn in chat only, and a 7000 character filter had dropped the long ones. The 644 surviving documents were repeated 39 times. That was the whole problem, not model capacity: a 15 minute warm restart with the first procedural trajectories already went to 6/10 and copied unseen domain names correctly.

Things that did not work, all measured on the same script with five fixed seeds and on the 19 server tests:

• Logit distillation from Qwen3.5-2B and from the 0.8B parent. Lower cross entropy on every held out set, and an agent that loops: after every tool result the student calls the same tool again and never answers. Measured on held out Nemotron trajectories, the parent puts <tool_call> as argmax after a tool result 87 percent of the time where the gold answer is a call 46 percent of the time, and it puts 100 percent of its mass on <think> at the start of every assistant turn. Masking the think tokens fixed the empty answers, not the looping. Distillation improves the language model and degrades the agent at this size. Dropped.
• A longer joint phase (17k instead of 11k steps). OCR improved, chat improved, tool calling collapsed to 4/10 and the MTP draft stopped matching the target. Text and images compete for 82M parameters.
• 8 layers instead of 6. Best language and OCR losses of the series, no gain on tool calling, which is decided by the data mix, not by capacity.
• Heavier shares of the hardest pattern (a gate on two quotes followed by heterogeneous tools). 4 percent of the mix passed the failing scenario in one run and broke single turn behavior everywhere else; 1 percent did neither.
• Anything evaluated with sampling. The script does not set a temperature; a 95M model sampled at 0.8 will drop a character of a 6 token domain name from time to time. A CI fixture has to be evaluated greedy, and the server tests are.

What worked: the procedural generator, Nemotron for real trajectories, system prompts in the synthetic data (the server tests always send one), harder OCR strings, whole document packing, one clean schedule from the pruned weights instead of warm restarts, one variable per run, and finally training on the CI shapes themselves. A fixture tests the server, not the model's intelligence: being predictable on what the CI asks is the point. The rule for maintainers is simple: add the shape of a new test to ci_shapes.py and rerun run.sh.

Results

Held out cross entropy on assistant tokens for this release: smoltalk 1.44, apigen 0.06, hermes multi turn 0.80, Nemotron 0.80, OCR 1.48. The general losses are a little worse than the best general checkpoint (smoltalk 1.37, hermes 0.61): the price of reliability on the CI shapes.

Reproduce

pip install --index-url https://download.pytorch.org/whl/cu128 --extra-index-url https://pypi.org/simple -r src/requirements.txt
export LLAMA_CPP=/path/to/llama.cpp
python3 dl.py                                                     # datasets
python3 prune_model.py --layers 0,11,14,19,22,23 --min-count 600  # -> ./pruned
./run.sh sft                                                      # render, synth, tokenize, text phase, joint phase
./ci.sh ckpt/sft-joint sft                                        # GGUF + function call script, greedy and seeds

Limitations

Chat is a toy, world knowledge is close to zero, OCR reads words it knows and guesses the rest, and copies of long rare strings can fail under sampling. Use greedy decoding. This model exists so that llama.cpp can test tool calling, vision and speculative decoding on every pull request with a 190 MB download.