N-Q-26M-Chat

A 26M-class Russian/English chat model trained from scratch on one V100, with exact tool calls

[!IMPORTANT] This is a research release, not a product. At 26M parameters the model is fluent in form but often factually wrong, repetitive under greedy decoding, and it does not solve multi-step word problems (GSM8K ≈ 2%). What it does reliably is call a tool: arithmetic, dates, units, sorting and the like are computed exactly by a fixed circuit, not by the network. See Honest results before drawing conclusions.

What this is

N-Q is the Q Project's take on MiniMind: a ladder of small Russian/English language models trained from scratch on a single 16GB V100, with the whole pipeline — tokenizer → pretrain → SFT → reasoning → LoRA → DPO → GRPO/PPO → agentic RL → distillation — in plain, readable PyTorch. This is the chat model of the smallest rung; the base model is N-Q-26M.

The architecture was chosen by an ablation, not assumed. At 26M and 0.5B tokens each (identical data, schedule and batch) we trained three candidates:

B won by 0.45 nats over A and ~0.2 over C at every checkpoint, and is the architecture of the whole ladder. Q's exact circuits are kept in every variant: the model writes <CALC>expr<EQ>, a deterministic executor writes the exact result and <ECALC>, and the model continues from there.

Honest results

An earlier version of our evaluation scored GSM8K at 16–20% because a numeric reference counted as correct when it appeared anywhere in the answer as a substring ("84" inside "184"). That was a scoring bug, not a capability; the numbers above use the final-number check only.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("q-project/N-Q-26M-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("q-project/N-Q-26M-Chat", trust_remote_code=True)

msgs = [{"role": "user", "content": "Сколько будет 4821 умножить на 37?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(ids, max_new_tokens=128, temperature=0.7, top_p=0.9, tokenizer=tok)   # tokenizer=tok switches the exact circuits on
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=False))
# <CALC>4821*37<EQ>178377<ECALC><ETHINK>4821 умножить на 37 равно 178377.<EOT>

• Pass tokenizer=tok. Without it there is no executor and the model has to "compute" the number after <EQ> itself — it will usually be wrong (4821*37 → 104621).
• Thinking: apply_chat_template(..., enable_thinking=True) opens a <THINK> block; leaving it unset (or False) lets the model decide. In this checkpoint False does not force a no-thinking answer — the SFT data never contained an empty <THINK><ETHINK> block, so forcing one is out of distribution.
• The model has its own generate() (recurrent Gated DeltaNet state + KV cache for the attention layers); it accepts temperature, top_p, top_k, repetition_penalty, max_new_tokens, stop_ids (default <EOT>, <eos>).

Packed single-file format (model.qpack)

The same QPACK1 container as Q-164M and Q-Omni: config, tokenizer, generation config, weights and a self-test in one mmap-friendly file (89 MiB vs 177 MiB safetensors).

• A repo or folder that has model.qpack and no *.safetensors loads with AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True); explicit: NQ4ForCausalLM.from_qpack("model.qpack").
• python qpack.py --selftest model.qpack prints SELFTEST 5/5 (top-1 and top-5 logits against the original fp32 model).
• Matrices are stored in fp16; 1-D tensors (norms, decay parameters) in fp32; the n-gram hash tables (prime sizes, offsets, multipliers) as exact int64 — a float round-trip would silently change the hashes. Greedy generation from model.qpack alone was identical to the safetensors weights on all prompts we checked.
• There is no torch-free runtime for this architecture yet (Q-Omni's qpack_run.py implements Q-Omni only).

Architecture

• 8 layers in the pattern LLLA ×2: three Gated DeltaNet linear-attention layers (3 key / 6 value heads, head dim 64, short causal conv k=4, sigmoid-gated RMSNorm output) then one gated attention layer (8 query / 2 KV heads, head dim 64, zero-centred QK-norm, partial RoPE 25%, θ=1e6, elementwise sigmoid output gate).
• hidden 512, SwiGLU MLP 512→704→512, Gated Residual with 4 parallel residual streams, zero-centred RMSNorm.
• Hashed n-gram embedding (PLE) injected after layer 2: 3-gram hashes into prime-sized tables, 16.8M parameters counted separately from the 29.7M backbone (the same accounting Qwen uses for its n-gram tables).
• Trained with Muon on the matrices + AdamW on the rest, WSD schedule, fp16 autocast on V100; torch.compile took training from 19k to 85k tokens/s.
• Context: 4096 positions in the config, trained at 1024.

Token layout

Chat format: <bos>[<SYSTEM>…<EOT>]<USER>…<EOT><MODEL>[<THINK>…<ETHINK>]answer<EOT>.

Training

GRPO, PPO and agentic-RL checkpoints were trained too (GSM8K / GSM8K-ru, verifiable rewards, circuits live during rollouts) and evaluated above; they are not released because they do not improve on this one at 26M.

Limitations

• Facts are frequently wrong ("the capital of France is…" goes off the rails); thin general knowledge; repetition under greedy decoding; long answers drift; weak creative writing.
• Does not solve multi-step math word problems; the circuits make single computations exact, not reasoning.
• The model sometimes ignores a correct circuit result when writing its final sentence (see the date example).
• enable_thinking=False is not a hard switch in this checkpoint (see Usage).
• No safety tuning. Research and education only.

License and data terms

Code and weights: Apache-2.0 (see LICENSE). Training data keeps its own terms — check each dataset card before commercial use: FineWeb-Edu (ODC-By 1.0), Cultura-Ru-Edu, Russian Wikipedia (CC BY-SA / GFDL), SmolTalk, GrandMaster-PRO-MAX, saiga_scored, saiga_preferences, russian-reasoning, UltraFeedback, GSM8K / GSM8K-ru.