Q-Omni-53M-Experimental

A 53M-parameter ternary decoder over one shared text / image / audio vocabulary

[!IMPORTANT] This is an experimental research release, not a product. It is small, fluent but often factually wrong, and its image/speech capabilities are weak or unusable — see Results below before using it for anything beyond experimentation.

Learn more: a full write-up of how it works is on the Q-Project blog, and you can run it live in your browser — no server, no Python, no sign-in.

Model summary

Q-Omni-53M-Experimental is a 53.2M-parameter decoder-only model developed by Q-Project. Its weights are ternary ({-1, 0, +1}, quantisation-aware from step one of training), and instead of a trained embedding table it uses a frozen 512-bit fingerprint per token, shared across one 49,280-token vocabulary that spans text, image tokens and audio tokens. A fixed circuit resolves arithmetic, dates and similar exact operations inside the generated token stream. The architecture is our own; it was pretrained from scratch on a single V100 GPU. General small-model ternary-quantisation research (in the spirit of BitNet b1.58) inspired the weight format.

Results

Architecture

• 7 layers, hidden 896, 14 query / 2 KV heads (head dim 64), plain 2-matrix SiLU MLP 896→2464→896, per-head QK-norm, per-head sigmoid output gate on attention, RoPE (θ=1e5), context 4096 (trained at 2048).
• Ternary weights from step 1: every attention/MLP projection is re-quantised to {-1,0,+1}·α (per-row absmean α) at each forward pass with a straight-through estimator.
• Frozen fingerprint vocabulary: each token is a fixed ±1 code of 512 bits instead of a trained embedding. Input = C[id]·W_in; readout = (h·W_out)·Cᵀ/√512 + frozen unigram log-prior. Ids ≥ 32768 (control, image, audio) carry an extra trainable correction on top of the frozen code.
• Exact circuits (circuits.py): the model writes <CALC>expr<EQ>; CircuitLogitsProcessor forces the exact result and <ECALC> — arithmetic, percentages, dates, weekday, unit conversion, comparison, sorting, letter/word counts.

Token layout

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("q-project/Q-Omni-53M-Experimental")   # adds <bos> automatically
model = AutoModelForCausalLM.from_pretrained("q-project/Q-Omni-53M-Experimental", trust_remote_code=True)
ids = tok("Once upon a time,", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True, top_k=40,
    suppress_tokens=list(range(32768, model.config.vocab_size)))[0], skip_special_tokens=True))

For exact circuits, build [<bos>, <USER>, *question_ids, <EOT>, <MODEL>] and generate with circuits.CircuitLogitsProcessor(tokenizer).

Packed single-file weights (model.qpack, 33 MB)

The same model also ships as one compact file: ternary weights at 1.6 bit/weight plus the fingerprint table, tokenizer, config and a self-test. This repo (folder with model.qpack and no other weights) loads the same way through AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True); explicitly: QOmniForCausalLM.from_qpack("model.qpack"). A torch-free numpy runtime is included (qpack_run.py): python qpack_run.py model.qpack "Once upon a time,", or --selftest to verify it reproduces the reference model exactly.

Limitations

• Facts are frequently wrong; thin general knowledge; repetitive under greedy decoding; weak creative writing. No safety tuning.
• Image generation does not produce recognisable pictures; speech recognition is unusable; speech synthesis was checked only numerically, not by listening.
• Ternary weights at this size are a research setting: an fp16 model of the same size was not compared for this checkpoint.
• Trained on one GPU with limited compute; do not expect competitive benchmark scores.

License and data terms

Code and weights: Apache-2.0 (see LICENSE). Training data keeps its own terms — check before commercial use: FineWeb-Edu and the SmolLM corpus (ODC-By 1.0 / Apache-2.0), CC3M (Google Conceptual Captions, research use), LibriSpeech (CC-BY 4.0), the Mimi audio codec (CC-BY 4.0) and the VQGAN image tokenizer (OpenRAIL++, use-based restrictions apply to that component).