gaggle-nanochat-pretrained-climbmix-20260924

A 3.3B-parameter decoder-only base language model, pre-trained from scratch with nanochat on one pass over ClimbMix (287B tokens). This is the causal model described in the pre-training section of Project Greenhouse: Progress Toward Fully Open and Sovereign Agentic Search (PDF). It has had no post-training (no instruction tuning or RLHF), so it continues text rather than following instructions. The chat model made from it with nanochat's midtraining + SFT recipe is castorini/gaggle-nanochat-ift-climbmix-20261031.

The same weights load four ways: with transformers (trust_remote_code=True), with vLLM (through a small plugin), with Ollama and llama.cpp (GGUF files, through a patch that adds the architecture to llama.cpp), and with nanochat's own code from the original checkpoint files, which are unchanged.

Model

Training

• Data: ClimbMix (karpathy/climbmix-400b-shuffle), one pass of 287B tokens. The last of its 6,543 shards is held out for validation. Documents are packed into 2,048-token sequences with nanochat's BOS-aligned best-fit packing.
• Optimizer and schedule: nanochat's, unchanged. Muon updates the weight matrices of the transformer blocks and AdamW the embeddings and scalar parameters; the learning rate warms up briefly, stays constant, and then decays linearly.
• Batch size: our only change. The batch grows from 262,144 to 2,097,152 tokens over training. Unlike nanochat, which scales its learning rates with the square root of the batch size, we keep their peak values unchanged as the batch grows.
• Compute: one node of eight NVIDIA H100 GPUs with FP8 matrix multiplications, about 1,600 H100-hours.
• This checkpoint: step 227,158, after 283B tokens. It is the checkpoint with the highest CORE measured during training; the final checkpoint (287B tokens) scored 0.004 lower.

Evaluation

Base-model benchmarks that need no instruction following: CORE (nanochat's average centered accuracy over 22 in-context learning tasks, 500 examples each), and 5-shot MMLU and MMLU-Pro scored by the log-likelihood of the answer letter with the LM Evaluation Harness.

All numbers are from our evaluation except nanochat d34's CORE, which is the published number.

Validation loss on the held-out ClimbMix shard is 0.596 bits per byte.

Usage

Sequences must start with the BOS token <|bos|>, as in training. The transformers tokenizer adds it for you (add_special_tokens=True, the default), and so do vLLM, llama.cpp and Ollama.

transformers

pip install "transformers>=4.57" "torch>=2.5"

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "castorini/gaggle-nanochat-pretrained-climbmix-20260924"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")   # starts with <|bos|>
out = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Output on one H200 GPU:

The capital of France is Paris. The capital of France is Paris. The capital of France is Paris. The capital of France

Log-likelihood of a continuation, as in the nanochat example below:

@torch.inference_mode()
def continuation_logprob(context, continuation):
    ctx = tokenizer(context)["input_ids"]                                    # starts with <|bos|>
    cont = tokenizer(continuation, add_special_tokens=False)["input_ids"]
    ids = torch.tensor([ctx + cont], device="cuda")
    logp = torch.log_softmax(model(ids).logits[0, len(ctx) - 1:-1], -1)
    return logp[torch.arange(len(cont)), torch.tensor(cont, device="cuda")].sum().item()

print(continuation_logprob("The capital of France is", " Paris"))    # -0.9538
print(continuation_logprob("The capital of France is", " Berlin"))   # -7.7336

These can differ from the nanochat example's printout in the last digits: nanochat runs attention with Flash Attention 3 on Hopper GPUs when that kernel is available, while this port always uses PyTorch's scaled_dot_product_attention, which is nanochat's own fallback and the kernel all our evaluations ran with. With the same kernel the two produce bit-identical logits (see below); across kernels, bf16 rounding moves log-probabilities slightly.

trust_remote_code=True is needed because the architecture is nanochat's rather than one transformers ships; configuration_gaggle.py and modeling_gaggle.py in this repo define it. They are plain PyTorch, import nothing beyond torch and transformers, and are short enough to read first. No nanochat checkout and no Flash Attention build are involved. model(input_ids).logits are fp32 and already soft-capped, as nanochat returns them; passing labels=input_ids gives the usual next-token loss. Batches may be padded on either side.

With the LM Evaluation Harness, pass add_bos_token=True (it does not add special tokens by default):

lm_eval --model hf --model_args pretrained=castorini/gaggle-nanochat-pretrained-climbmix-20260924,trust_remote_code=True,dtype=bfloat16,add_bos_token=True --tasks mmlu --num_fewshot 5

vLLM

vLLM has no built-in class for this architecture, so install the small plugin that provides one; vLLM then finds it automatically:

pip install vllm https://huggingface.co/castorini/gaggle-nanochat-pretrained-climbmix-20260924/resolve/main/vllm_gaggle-0.1.0-py3-none-any.whl
vllm serve castorini/gaggle-nanochat-pretrained-climbmix-20260924 --trust-remote-code

from vllm import LLM, SamplingParams

llm = LLM("castorini/gaggle-nanochat-pretrained-climbmix-20260924", trust_remote_code=True)
out = llm.generate(["The capital of France is"], SamplingParams(max_tokens=20, temperature=0))
print(out[0].outputs[0].text)

Output on one H200 GPU:

 Paris. The capital of France is Paris. The capital of France is Paris. The capital of France

The plugin (vllm_plugin/ in this repo; one model file of about 230 lines) registers the architecture through vLLM's vllm.general_plugins entry point, so every vLLM process finds it; nothing in vLLM itself is changed. It reads the same config.json and safetensors as transformers. Every nanochat component maps onto ordinary tensor operations except the embedding smear, which mixes the previous token's embedding into each position: while decoding, vLLM gives the model only the newest token, so the port recovers the previous embedding from the cache with one extra attention layer whose two-token window is set up to select the previous position exactly. It runs on one GPU (no tensor parallelism) with the model's 2,048-token context, with or without CUDA graphs and torch.compile. On one H200, 256 prompts x 256 new tokens run at about 13,000 tokens/s, three times transformers generate() on the same batch.

Ollama and llama.cpp

Ollama runs models through llama.cpp, which has no class for this architecture, so both need gaggle-llama.cpp-b11669.patch, which adds one (a model file, src/models/gaggle.cpp, and the GGUF converter; no existing model changes). It is written against llama.cpp b11669, the version Ollama v0.40.5 pins. Stock Ollama and llama.cpp cannot load these GGUF files.

Build Ollama v0.40.5 with the patch. Ollama applies every patch in llama/compat/ to the llama.cpp source it downloads, so the patch goes there. You need Go, CMake 3.24 or newer and a C/C++ compiler; Ollama's docs/development.md has the details, including GPU backends.

git clone --depth 1 --branch v0.40.5 https://github.com/ollama/ollama.git && cd ollama
curl -L -o llama/compat/002-gaggle.patch \
  https://huggingface.co/castorini/gaggle-nanochat-pretrained-climbmix-20260924/resolve/main/gaggle-llama.cpp-b11669.patch
cmake -B build . && cmake --build build --parallel 8
./ollama serve

Then, in another shell, download a GGUF file and the Modelfile and create the model:

mkdir gaggle && cd gaggle
R=https://huggingface.co/castorini/gaggle-nanochat-pretrained-climbmix-20260924/resolve/main
curl -L -O $R/Modelfile -O $R/gaggle-nanochat-pretrained-climbmix-20260924-Q8_0.gguf
/path/to/ollama/ollama create gaggle -f Modelfile
/path/to/ollama/ollama run gaggle "The capital of France is"

The Modelfile passes prompts through unchanged (this is a base model) and sets the context to 2,048 tokens. The patched Ollama can also fetch a GGUF file from the Hub by itself, with no Modelfile: ollama run hf.co/castorini/gaggle-nanochat-pretrained-climbmix-20260924:Q8_0 (or :BF16, :Q4_K_M). Prompts again pass through unchanged, but the context is Ollama's default; set it with /set parameter num_ctx 2048 in the session or "num_ctx": 2048 in the API options.

Greedy decoding through the API:

curl http://localhost:11434/api/generate -d '{"model": "gaggle", "prompt": "The capital of France is", "raw": true, "stream": false, "options": {"temperature": 0, "num_predict": 20}}'

The response field, on CPU:

 Paris. The capital of France is Paris. The capital of France is Paris. The capital of France

Three GGUF files are provided. We compared each with transformers in fp32 over 1,718 next-token predictions on four texts (prose, code, mixed scripts and 1,600 tokens of this card), running llama.cpp on CPU with the f16 KV cache Ollama uses:

The Q4_K_M file keeps ffn_down at 8 bits (llama-quantize --tensor-type ffn_down=q8_0 ... Q4_K_M), which halves its KL relative to a standard Q4_K_M (0.020 against 0.047 on the 1,600-token text). There is no F16 file: some activations exceed float16's range, and an F16 GGUF gives NaN logits.

To use llama.cpp directly, apply the patch to b11669:

git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp && git checkout b11669
curl -L https://huggingface.co/castorini/gaggle-nanochat-pretrained-climbmix-20260924/resolve/main/gaggle-llama.cpp-b11669.patch | git apply
cmake -B build && cmake --build build -j 8 --target llama-server    # add -DGGML_CUDA=ON for NVIDIA GPUs
build/bin/llama-server -m /path/to/gaggle-nanochat-pretrained-climbmix-20260924-Q8_0.gguf -c 2048

The patched tree's convert_hf_to_gguf.py converts the safetensors in this repo; the BF16 and Q8_0 files here were made with it (--outtype bf16, --outtype q8_0) and the Q4_K_M file from the BF16 one with llama-quantize.

Notes:

• Every component maps onto existing llama.cpp operations except the embedding smear (see the vLLM notes above). The port keeps the previous token's embedding as a one-token recurrent state, so llama.cpp treats the model like its hybrid attention/recurrent models: prompts are cached and reused, but the context cannot be shifted, and generation stops when the 2,048-token context is full. Ollama shortens longer prompts itself.
• The short-window layers attend to 512 earlier tokens; the other layers are exact up to 2,049 tokens. Keep the context at 2,048 (num_ctx in the Modelfile, -c for llama.cpp).
• nanochat's rotary embedding turns the opposite way from llama.cpp's. The converter negates the second half of each query and key head, which the QK norm and the attention product cancel exactly.
• We tested Ollama and llama.cpp on CPU, and llama.cpp with CUDA 12.8 on an NVIDIA RTX PRO 6000 Blackwell, where llama-server generates about 190 tokens/s from the BF16 file. Ollama's GPU builds compile the same code but were not tested, nor were Apple Metal and other backends.

nanochat

The original training checkpoint is still here, unchanged (model_227158.pt, meta_227158.json, tokenizer/), so nanochat's own code loads it exactly as before. Set up nanochat at the commit the model was trained with:

git clone https://github.com/karpathy/nanochat.git
cd nanochat && git checkout 92d63d4
uv sync --extra gpu        # use --extra cpu on a machine without a CUDA GPU

Save the following as example.py in the nanochat repository root (nanochat is imported from there, not installed as a package) and run uv run --extra gpu --with huggingface_hub python example.py.

import os, torch
from huggingface_hub import snapshot_download

# only the nanochat files; the safetensors are for transformers and vLLM
path = snapshot_download("castorini/gaggle-nanochat-pretrained-climbmix-20260924",
                         allow_patterns=["model_227158.pt", "meta_227158.json", "tokenizer/*"])
os.environ["NANOCHAT_BASE_DIR"] = path   # nanochat loads the tokenizer from $NANOCHAT_BASE_DIR/tokenizer

from nanochat.checkpoint_manager import build_model

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model, tokenizer, meta = build_model(path, step=227158, device=device, phase="eval")

# Greedy continuation
prompt = "The capital of France is"
ids = tokenizer.encode(prompt, prepend=tokenizer.get_bos_token_id())
new = list(model.generate(ids, max_tokens=20, temperature=0.0))
print(prompt + tokenizer.decode(new))

# Log-likelihood of a continuation given a context
@torch.inference_mode()
def continuation_logprob(context, continuation):
    ctx = tokenizer.encode(context, prepend=tokenizer.get_bos_token_id())
    cont = tokenizer.encode(continuation)
    x = torch.tensor([ctx + cont[:-1]], device=device)
    y = torch.tensor([[-1] * (len(ctx) - 1) + cont], device=device)   # -1 = ignore context positions
    return -model(x, targets=y, loss_reduction="sum").item()

print(continuation_logprob("The capital of France is", " Paris"))
print(continuation_logprob("The capital of France is", " Berlin"))

Output on one H200 GPU:

The capital of France is Paris. The capital of France is Paris. The capital of France is Paris. The capital of France
-0.9807320833206177
-7.733334541320801

Notes:

• On GPUs with bfloat16 support the model computes in bfloat16; on CPU and MPS it uses float32. Override with NANOCHAT_DTYPE=bfloat16|float16|float32.
• For further pre-training or fine-tuning, use nanochat's training scripts (scripts/base_train.py, scripts/chat_sft.py) with this checkpoint as the starting point.

The implementations agree

transformers, vLLM and nanochat load the same tensors. We checked them against nanochat's own forward pass on the original checkpoint, which is what every number in the Evaluation section was computed with, and the llama.cpp port (which Ollama runs) against transformers.

In bf16, any change of kernel shape — batching, padding, a KV cache, a different attention kernel — moves logits by rounding, which can flip near-tied greedy choices after a few dozen tokens; nanochat itself behaves the same way. convert_nanochat_to_hf.py is the converter that produced the safetensors from model_227158.pt.

Files

• model-0000{1,2,3}-of-00003.safetensors, model.safetensors.index.json: the weights for transformers and vLLM. These are the tensors of model_227158.pt, unchanged in value and dtype (fp32 matrices, bf16 embedding tables), renamed to model.<nanochat name> (lm_head.weight keeps its name).
• config.json, generation_config.json, configuration_gaggle.py, modeling_gaggle.py: the architecture for transformers (and the configuration vLLM reads).
• tokenizer.json, tokenizer_config.json: the tokenizer, converted from tokenizer/tokenizer.pkl; it encodes every text exactly as nanochat does and prepends <|bos|>.
• model_227158.pt: model weights (nanochat state dict), for nanochat.
• meta_227158.json: model configuration, training configuration, and validation loss.
• tokenizer/: the 32,768-entry tokenizer in nanochat's format (tokenizer.pkl, token_bytes.pt).
• gaggle-nanochat-pretrained-climbmix-20260924-{BF16,Q8_0,Q4_K_M}.gguf: the model for Ollama and llama.cpp.
• Modelfile: the Ollama model definition (prompts unchanged, 2,048-token context).
• gaggle-llama.cpp-b11669.patch: adds the architecture to llama.cpp b11669, for Ollama v0.40.5 and llama.cpp.
• SHA256SUMS: checksums of the weight and metadata files, the GGUF files, the patch and the Modelfile.

Optimizer state is not included.

Related

• castorini/gaggle-nanochat-ift-climbmix-20261031: the chat model made from this checkpoint with nanochat's midtraining + SFT recipe (no RL); loads in transformers, vLLM and nanochat like this repo.
• castorini/gaggle-nanochat-pretrained-climbmix-bidirectional-20260929: the bidirectional variant, converted from this run's causal checkpoint at 245B tokens and trained on the remaining 42B tokens with bidirectional attention and masked next-token prediction, following LLM2Vec.
• castorini/gaggle-reranker-20261005: the pointwise reranker fine-tuned from this checkpoint.

Citation

If you use this checkpoint, please cite our paper:

@misc{lin2026projectgreenhouseprogressfully,
  title={{Project Greenhouse}: Progress Toward Fully Open and Sovereign Agentic Search},
  author={Jimmy Lin and Sahel Sharifymoghaddam and Lingwei Gu and Nour Jedidi},
  year={2026},
  eprint={2610.11922},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2610.11922}
}