Boris-1.7-D60M-n30M

A 60M-parameter dense decoder-only transformer with a 30M-parameter n-gram embedding table spliced into the residual stream after block 2. One experiment, one variable: does 30M of lookup buy what 30M of dense weight would, at a fixed 2.5B-token budget on a single RTX 3060 12GB?

The trunk is deliberately conventional — the table is the only thing under test.

Model

The table is keyed on the literal last 2 and 3 token ids — an explicit top-K vocabulary (281,250 bigram + 187,500 trigram rows, one shared OOV row), built by ngram.py over the corpus. No hashing, exact searchsorted lookup, one gather per position and no matmul, so compute per token barely moves.

Training

• 2.5B tokens: Ultra-FineWeb bulk, then cosmopedia-v2 + fineweb-edu anneal over the final 15%
• WSD schedule (2% warmup, 85% stable, 1-sqrt decay), Muon on 2-D linear maps, AdamW on everything else, bf16 + torch.compile
• 10,172 steps, 245,760 tokens/step, 22h37m wall clock
• Final val: loss 3.235, ppl 25.4, 1.0012 bits/char; n-gram hit rates ~65% bigram / ~21% trigram

Resumable to the byte (optimiser states, data cursor, all RNGs round-trip through the checkpoint).

Results (zero-shot)

Scored with the model's own tokenizer; harness validated against GPT-Neo-125M's published numbers first. --no-ngram ablates the table at inference, which measures how much the finished model leans on it — not the dense control (that needs its own training run).

Ablating the table costs 0.069 bits/char and 0.7–3.1 points across the six tasks. The two ARC splits move most; HellaSwag and WinoGrande barely notice.

Additional benchmarks, table intact only:

Usage

This is not a transformers checkpoint — it's a custom architecture (60M dense trunk + 30M n-gram lookup table) with a custom byte-level BPE tokenizer. Load it with the loader in this repo:

from loader import load_model, load_tokenizer

model = load_model(".").eval()     # reads config.json + model.safetensors
tok = load_tokenizer(".")          # reads tokenizer.json (custom 32,768 BPE)

The n-gram table and its top-K lookup arrays are inside model.safetensors, so that one file — plus config.json — is enough to run the full model. lm_head.weight is tied to the input embedding and stored once.

Tokenize with the tokenizer's GPT-2-style byte-level split (digit runs are capped at three characters):

ids = tok.encode("The capital of France is")

Generation loop (greedy, single token step):

import torch

def generate(prompt, n_tokens=32):
    ids = tok.encode(prompt)
    for _ in range(n_tokens):
        with torch.no_grad():
            logits = model(torch.tensor([ids]))[0]
        nxt = int(logits[0, -1].argmax())
        ids.append(nxt)
        if nxt == 0:            # <|endoftext|>
            break
    return tok.decode(ids)

print(generate("The capital of France is"))

AutoTokenizer also works (self-contained tokenizer.json), but the model itself has no transformers class:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("opencerebral/Boris-1.7-D60M-n30M")

Run python loader.py "The capital of France is" for a next-token smoke test.

Copyright & License

Copyright 2026 OpenCerebral

This project and all associated files (the "Work") are licensed under the Apache License, Version 2.0 (the "License"); you may not use this project except in compliance with the License. You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.