N-Q-164M-Chat

A 164M-class Russian/English chat model trained from scratch on one V100, with exact tool calls

[!IMPORTANT] This is a research release, not a product. At 164M parameters the model writes fluent, structured answers in Russian and English and calls its calculator / date / unit tools reliably, but it still states wrong facts confidently and solves only a few multi-step word problems (GSM8K ≈ 7%). See Honest results.

What this is

N-Q is the Q Project's take on MiniMind: a ladder of small Russian/English models trained from scratch on a single 16GB V100, with the whole pipeline — tokenizer → pretrain → SFT → reasoning → DPO → GRPO → agentic RL — in plain PyTorch. This is the chat model of the 164M rung (base: N-Q-164M); it went through exactly the same recipe as N-Q-26M-Chat.

Q's exact circuits are part of the model: it writes <CALC>expr<EQ>, a deterministic executor writes the exact result and <ECALC>, and the model continues from there.

Honest results

Every stage of the recipe was evaluated with the same scripts; the released checkpoint is the last one (agentic RL), because at this size each RL stage helped a little instead of doing nothing.

GSM8K is scored by the final number only (thinking on). ±1.1 pp is one standard error at these rates; the jump from reasoning SFT to the final model is real, the step-to-step differences are partly noise. The multi-turn agentic episodes themselves stayed close to zero reward — the gain comes mostly from better single-turn tool use.

Samples (temperature 0.7, from the evaluation log):

Сколько будет 347 умножить на 86? — <THINK><CALC>347*86<EQ>29842<ECALC><ETHINK>347 умножить на 86 равно 29842.

Товар стоит 1250 рублей, скидка 15%. Сколько платить? — <CALC>1250*15/100<EQ>187.5<ECALC><CALC>1250-187.5<EQ>1062.5<ECALC> Скидка 187.5 руб., итоговая цена 1062.5 руб.

How many days are between March 3, 2026 and July 19, 2026? — <CALC>dur:2026-03-03,2026-07-19<EQ>138<ECALC> 138 days.

What is the capital of France? — The capital of France is Paris. It is located in the northern part of the country and is known for its historical landmarks, such as the Tuileries Gardens and the Eiffel Tower.

Объясни простыми словами, что такое фотосинтез. — Фотосинтез — это процесс, в ходе которого растения … преобразуют световую энергию солнца в химическую энергию, используя углекислый газ и воду. … выделяет кислород (CO2) … (right idea, wrong detail: the typical failure mode at this size)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("q-project/N-Q-164M-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("q-project/N-Q-164M-Chat", trust_remote_code=True)

msgs = [{"role": "user", "content": "Товар стоит 3200 рублей, скидка 25%. Какая итоговая цена?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(ids, max_new_tokens=160, temperature=0.7, top_p=0.9, tokenizer=tok)   # tokenizer=tok switches the exact circuits on
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=False))
# ... Скидка 800 руб., итоговая цена 2400 руб.<EOT>

• Pass tokenizer=tok. Without it there is no executor and the model has to "compute" the number after <EQ> itself, which is usually wrong.
• Thinking: apply_chat_template(..., enable_thinking=True) opens a <THINK> block; leaving it unset lets the model decide (it thinks on calculations and word problems, answers chit-chat directly).
• The model has its own generate() (recurrent Gated DeltaNet state + KV cache for the attention layers); it accepts temperature, top_p, top_k, repetition_penalty, max_new_tokens, stop_ids (default <EOT>, <eos>).
• model.qpack / model.q8.qpack: single-file packed weights (fp16 / int8), loadable with from_pretrained(..., trust_remote_code=True) from a folder without *.safetensors and by the qpack C++ engine.

Architecture

• 16 layers, pattern LLLA ×4: three Gated DeltaNet linear-attention layers (4 key / 8 value heads, head dim 112, conv k=4, gated RMSNorm output) then one gated attention layer (14 query / 2 KV heads, head dim 64, zero-centred QK-norm, partial RoPE 25%, θ=1e6, sigmoid output gate).
• hidden 896, SwiGLU 896→1664→896, Gated Residual with 4 residual streams, zero-centred RMSNorm.
• Hashed n-gram embedding (PLE): 58.7M parameters counted separately from the 166.8M backbone.
• Tokenizer: 8,256 tokens (8,192 byte-level BPE RU+EN with single-digit splitting + 64 control tokens), shared with N-Q-26M.
• Chat format: <bos>[<SYSTEM>…<EOT>]<USER>…<EOT><MODEL>[<THINK>…<ETHINK>]answer<EOT>.

Training

All on one V100 16GB.

Limitations

• Facts are often wrong in the details; general knowledge is thin; long answers drift; creative writing is weak (fluent verse without meaning).
• Multi-step math word problems: ~7%. The circuits make single computations exact, not the plan.
• Identity is shaky ("Я программист…" when asked who it is).
• No safety tuning. Research and education only.

License and data terms

Code and weights: Apache-2.0 (see LICENSE). Training data keeps its own terms — check each dataset card before commercial use: FineWeb-Edu (ODC-By 1.0), Cultura-Ru-Edu, Russian Wikipedia (CC BY-SA / GFDL), SmolTalk, GrandMaster-PRO-MAX, saiga_scored, saiga_preferences, russian-reasoning, UltraFeedback, GSM8K (MIT) / GSM8K-ru.