cagliostro-v3.5

The chat version is out: cagliostro-v3.5-chat, this model after one epoch of supervised fine-tuning on smol-smoltalk. It ends its own turns, follows system prompts, and comes out level with SmolLM2-135M-Instruct on MT-Bench.

A 146M parameter decoder-only language model. It is cagliostro-v3's final checkpoint with one more short training stage: 0.39B tokens of v3's own cooldown data mixed with how-to and textbook text from Cosmopedia. Same architecture, same tokenizer, 75.39B tokens in total.

It scores 27.49 on the Open SLM Leaderboard metric, above every model listed there at the time of release.

Results

Zero-shot, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script, on the float32 weights in this repository.

The Index is the leaderboard's own formula, (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) and CombinedARC is the mean of ARC-Easy and ARC-Challenge.

For context, using the leaderboard's published figures for every other model:

The lead over SmolLM2-135M is 0.36 Index on the leaderboard's figures. Re-evaluated in the same harness, SmolLM2-135M scores 27.01 and cagliostro-v3.5 is ahead by 0.48, but a paired bootstrap over benchmark items puts the 95% interval for that gap at [-0.82, +1.75]. On points it is first. On these five benchmarks the two models cannot be told apart with confidence, and we would not claim more than that.

cagliostro-v3 appears in the same-harness charts as it measures today, Index 26.31. Its leaderboard row lists 26.55.

What changed from v3

The bottom six rows are v3's cooldown mixture scaled to 55%. The WikiHow set is 174M tokens, so it was seen about 0.7 times. The textbook slice is the first 150M tokens of those three subsets.

The gain is mostly ArithMark-3 (+2.6 points) and HellaSwag (+0.9), with smaller gains on ARC-Challenge and PIQA. ARC-Easy moved the other way by about a point. That drop also shows up when v3 is trained the same way on its own cooldown mixture, so it comes from the extra low learning rate stage rather than from the new data.

How it was chosen

Six runs started from the same v3 checkpoint, identical except for the data. Before any of them finished we wrote down what a winner had to do: score above SmolLM2-135M's 27.13, beat SmolLM2-135M re-measured in the same harness, and beat v3 with the paired 95% interval above zero. Then a second run of the same recipe with a different data seed had to clear 27.13 as well.

WikiHow alone passed once and then failed its second seed, so it was dropped. WikiHow with textbooks passed on both seeds. This repository holds the seed 2 checkpoint, the stronger of the two. The recipe averages 27.40 over both.

Chat data lost points, as it does for most base and instruct pairs on the leaderboard. The benchmarks score raw log-likelihood with no chat template, so text that looks like a conversation does not help.

Contamination checks

No benchmark training, validation or test split was used at any point.

Every new document went through the same 13-gram decontamination v3 used, against HellaSwag validation, ARC test, PIQA validation and ArithMark-3, dropping any document with a match. In a sample of 8,000 documents from the two Cosmopedia sources it dropped none.

The HellaSwag gain deserves a closer look, because about two thirds of HellaSwag is built from real WikiHow articles and Cosmopedia's WikiHow set was generated from real WikiHow titles.

19.3% of HellaSwag's WikiHow validation items share an article title with the Cosmopedia set. For the training split the figure is 18.5%, so the validation items are not singled out. The gain is about the same on items whose title appears in the training data (+1.45) as on items whose title does not (+1.24). ActivityNet items, which have nothing to do with WikiHow, barely moved (+0.09). That pattern is a model getting better at everyday how-to text, not one that has seen the test articles.

Sample outputs

Greedy decoding, float32, first sentence of each completion, unedited. These are all twelve prompts that were tried; the video shows four of them. The full completions are in samples.json.

The misses are typical for this size. It adds 47 and 68 correctly and then divides by 2 for no reason, gets the train's speed wrong, starts boiling an egg by cracking it, and describes a shoelace as yarn.

Model details

The architecture is defined in this repository. trust_remote_code=True is required because CagliostroForCausalLM is not part of transformers.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bench-labs/cagliostro-v3.5"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)

ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

This is a base model with no instruction tuning and no chat template. It completes text. For chat, use cagliostro-v3.5-chat.

Fast inference

generate() now uses a KV cache, so it is fast out of the box. On a GPU, generate_fast() goes further with a static cache and CUDA graphs:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bench-labs/cagliostro-v3.5"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float16).to("cuda")

ids = tok("The water cycle begins when", return_tensors="pt").input_ids.to("cuda")
out = model.generate_fast(ids, max_new_tokens=200, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

Speed of cagliostro-v3.5-chat, which shares this architecture and this code:

Measured with the code in this repository. GPU numbers use 256 new tokens; CPU numbers use 128. Both use a 23-token prompt and repetition penalty 1.1, median of three runs. On a GPU, generate_fast() uses CUDA graphs; on a CPU it falls back to a static-cache loop.

• The forward pass used for benchmark scoring is unchanged bit for bit, so leaderboard numbers do not move.
• In float32, generate() and generate_fast() give exactly the same greedy text as the previous release on all 12 sample prompts.
• float16 and bfloat16 are about 1.7x faster than float32. Greedy text can drift from float32 at a near tie, usually after a few dozen tokens (float16: 8 of 12 prompts identical over 200 tokens). float16 tracked float32 slightly better than bfloat16.
• compile=True fuses the decode step with torch.compile for about another 1.5x. It needs a working Triton install and costs about 40 seconds the first time in each process.
• generate_fast() takes one sequence at a time and supports temperature, top_p, seed, repetition_penalty and a transformers TextStreamer. Use generate() for batches.

Evaluate in float32 as described below. These speed options are for generation only.

Reproducing the evaluation

pip install lm-eval
python -m lm_eval --model hf \
  --model_args pretrained=bench-labs/cagliostro-v3.5,dtype=float32,trust_remote_code=True \
  --tasks hellaswag,arc_easy,arc_challenge,piqa \
  --num_fewshot 0 --batch_size 64 --device cuda:0

ArithMark-3 uses the script linked from the leaderboard, pointed at the same model id with --dtype float32. Evaluate in float32. A bfloat16 round trip moves logits enough at this logit cap to change borderline multiple-choice answers.

These exact commands, run against this repository, return the numbers in the results table.

Limitations

English only. 2,048 token context. No instruction tuning (see cagliostro-v3.5-chat for that), no safety tuning, no RLHF. At 146M parameters it confabulates freely and should not be relied on for factual questions. ARC-Easy is about a point below cagliostro-v3. The mathematics ability measured by ArithMark is arithmetic and short symbolic work, not general mathematical reasoning.

Beyond the leaderboard

The leaderboard reports five benchmarks, so we also ran a wider set against SmolLM2-135M, the model this one replaced at the top. Both sides got the same setup: lm-evaluation-harness 0.4.13, zero-shot, float32, full test sets, 2,048-token context. The first four rows reproduce the leaderboard's own numbers exactly. Differences are cagliostro-v3.5 minus SmolLM2-135M, with a 95% interval from a paired bootstrap over items.

Bold marks an interval that excludes zero.

Outside the leaderboard, SmolLM2-135M is the stronger general language model. It is clearly ahead on plain language modelling (WikiText, LAMBADA), grammar (BLiMP), science questions (SciQ) and ARC-Easy. The two are statistically level on the other nine tasks; CommonsenseQA and MMLU are at chance for both. This model's lead on the board comes from arithmetic, which the Index weights at 0.65: 6.5 points on ArithMark-3 and 26.6 on ArithMark-2, a separate arithmetic set the board does not use. SmolLM2-135M was trained on 2 trillion tokens, this model on 75.4 billion.

The chat versions were compared too. On MT-Bench, judged pairwise by gpt-oss-120b in both answer orders, cagliostro-v3.5-chat won 66 question-turns to SmolLM2-135M-Instruct's 55, with 39 ties (sign test p = 0.36, so level). Full tables, every answer, all 320 judge verdicts and the scripts are in the chat repo's evals/ folder and card.

License

Apache-2.0. The training data is drawn from FineWeb-Edu and FineMath (ODC-By), DCLM-Baseline and OpenMathInstruct-2 (CC-BY-4.0), Cosmopedia v1, Cosmopedia v2 and SmolTalk (Apache-2.0).