N-Q-164M

A 164M-class Russian/English base model: mini Qwen3.8-Flash-Next, pretrained from scratch on one V100

[!IMPORTANT] This is a research release, not a product. A plain text-continuation model: fluent Russian and English, much better grounded than N-Q-26M, still inventing facts. For dialogue and exact tool calls use N-Q-164M-Chat.

Model summary

N-Q-164M is the base model of the largest dense rung of N-Q, the Q Project's MiniMind-style ladder of small RU/EN models trained from scratch on a single 16GB V100. The architecture is the same one that won the 26M ablation (see the N-Q-26M-Chat card), scaled up: Gated DeltaNet linear attention interleaved 3:1 with gated full attention, a 4-stream Gated Residual and a hashed n-gram embedding.

166.8M backbone parameters + 58.7M n-gram table (225.5M stored), the same 8,256-token RU+EN byte-level BPE as N-Q-26M, pretrained on 8.0B tokens (30,517 steps × 256×1024 tokens, WSD schedule, Muon + AdamW, torch.compile, ~19k tokens/s, ≈117 V100-hours). 5% of rows were dialogues and <CALC> circuit rows from step 1, so the base model already knows the chat and tool formats.

Results (final checkpoint, step 30,517)

Perplexities are per token of this tokenizer and are not comparable to models with other tokenizers.

Samples (temperature 0.7):

Photosynthesis is the process by which plants produce food from the sun. Photosynthesis occurs in the chloroplasts of plants… In photosynthesis, chlorophyll is the pigment that gives the plant its green color.

Однажды в Петербурге, в честь 100-летия со дня рождения Александра Сергеевича Пушкина, … состоялось торжественное открытие памятника Александру Сергеевичу Пушкину.

Москва — столица России. Центр Московской области. Расположена на юго-западе области, в 440 км к востоку от Москвы… (fluent, still drifts into invented geography)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("q-project/N-Q-164M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("q-project/N-Q-164M", trust_remote_code=True)
ids = tok("Москва — столица", return_tensors="pt")
out = model.generate(ids, max_new_tokens=80, temperature=0.7, top_p=0.9, stop_ids=(2,))
print(tok.decode(out[0], skip_special_tokens=True))

The model has its own generate() (recurrent Gated DeltaNet state + KV cache). Pass tokenizer=tok to switch on the exact <CALC> circuits if you prompt it in the chat format.

Packed single-file formats

model.qpack (QPACK1: fp16 matrices, fp32 1-D tensors, exact int64 n-gram hash tables) and model.q8.qpack (QPACK2, int8 weights, see QPACK2.md) hold weights, config, tokenizer and a self-test in one mmap-friendly file. A folder or repo with a *.qpack and no *.safetensors loads through AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True); python qpack.py --selftest model.qpack checks top-1/top-5 logits against the fp32 model.

Architecture

• 16 layers, pattern LLLA ×4: Gated DeltaNet (4 key / 8 value heads, head dim 112, conv k=4) ×3, then gated attention (14 query / 2 KV heads, head dim 64, zero-centred QK-norm, partial RoPE 25% θ=1e6, sigmoid output gate).
• hidden 896, SwiGLU 896→1664→896, Gated Residual with 4 residual streams, zero-centred RMSNorm, untied output head.
• Hashed 3-gram embedding (PLE): 58.7M parameters in prime-sized tables, counted separately from the backbone.
• Token layout: ids 0–8191 text (0 <pad>, 1 <bos>, 2 <eos>), 8192–8255 control tokens (<USER> <MODEL> <EOT> <THINK> <ETHINK> <CALC> <EQ> <ECALC> <NEED> <QUOTE> <SYSTEM>, reserved <R11>…<R63>).
• Context: 4096 positions in the config, trained at 1024.

Limitations

• Facts are often wrong; general knowledge is thin for its size class; repetitive under greedy decoding.
• Base model: no instruction following beyond the 5% dialogue rows it saw in pretraining.
• No safety tuning. Research and education only.

License and data terms

Code and weights: Apache-2.0 (see LICENSE). Training data keeps its own terms — check before commercial use: FineWeb-Edu (ODC-By 1.0), Cultura-Ru-Edu, Russian Wikipedia 20231101.ru (CC BY-SA / GFDL); dialogue/circuit rows from SmolTalk, GrandMaster-PRO-MAX, saiga_scored and our own <CALC> corpus.