Ant 15M site QA

Ant 15M (G28) · Qwen 3.8-Flash Policy Optimized (87.5% Site QA)

A 28-layer thin hybrid model (AMMMMMAMMMMMAMMMMMAMMMMAMMMA, d = 192, 6 sparse attention + 22 Mamba2 layers), fine-tuned with a co-executed Qwen 3.8-Flash policy-actor loop on portfolio retrieval prompts. It reaches 21/24 (87.5%) factual verification accuracy on the portfolio's benchmark questions, with 13/13 (100%) calibrated safe refusal on out-of-scope queries (0 wrong answers shown on unanswerable questions), and answers evidence-first in under 20 MB (19.4 MB dynamic int8 with SmoothQuant).

A 14.9M-parameter hybrid (attention + Mamba2) decoder trained from scratch to answer a visitor's question from passages a website retrieves, and to refuse when the passages don't say. It runs in the browser on onnxruntime-web's WASM backend (no WebGPU needed): one 21 MB int8 file.

It is the successor to Tiny 30M: half the parameters and download, faster per token, and on the site's own questions as good or better. It runs the opt-in on-device chat on mihirkarbelkar.com.

Architecture

Eight layers alternating attention and Mamba2, AMAMAMAM, d = 320, each followed by a SwiGLU MLP (hidden 704), pre-norm RMSNorm, tied 16K byte-level BPE embedding.

• A: grouped-query attention, 8 query / 2 key-value heads of 64, QK-norm, RoPE.
• M: Mamba2 (SSD), expand 2 (640 inner), 10 heads of 64, state 64, conv 4.

It came out of a 15M architecture search (eight layer patterns at equal size and data: attention only, attention bookends around GRU / Mamba2 / Gated DeltaNet / RWKV-7, and mixes). Interleaving attention with Mamba2 won: the attention layers pick the passage that holds the answer and copy from it, the Mamba2 layers carry the decision whether the passages answer at all.

Use

Prompt token ids: [7] + tokens(sources) + [3] + tokens(question) + [2], where sources is

SOURCE 1: <page title>
<passage text>

SOURCE 2: …

(≈ 2,000 characters of passages works best). Generation ends at token 1. The model answers evidence first:

Source 2: "an exact span copied from source 2"
Answer: one or two plain sentences.

Source: none + Answer: The context does not say. is a refusal, and Chat: … is a reply to small talk that states no facts.

decoder.int8.onnx is a single graph used for the prompt and each new token:

• inputs: input_ids int64 (1, T) and one state tensor per entry in meta.json's states
• outputs: logits (1, 16384) for the last position, then new_<state> for every state

Per attention layer the state is a key/value cache kvN of shape (2, 1, 2, 1+P, 64) that grows by T each call (its first slot is a dummy the graph masks out); per Mamba2 layer it is a conv window convN (1, 768, 3) and an SSM state ssmN (1, 10, 64, 64), fixed size. Start from zeros, then feed each call's outputs back in. The prompt can be read in one call or in chunks (the states carry over exactly); chunks of ~256 tokens keep Mamba2's prompt form cheap on long prompts.

Recommended decoding. Score the probability of Source: none after the prompt and refuse when it is at least 0.3. Otherwise decode greedily (repetition penalty 1.1), banning : right after a first token Source so the model must cite a source number. Check that the quote really occurs in the cited source before showing the answer; if it doesn't, show the sources instead. A reference implementation is the portfolio's src/workers/tiny.worker.ts.

Training

• Pretraining: 2.9B tokens of FineWeb-Edu (60%) and FineWeb (40%, biased towards structured pages), seq 1024, 262K tokens per step, AdamW lr 3e-3 with cosine decay. Held-out loss 3.39 nats/token.
• Fine-tuning: the same ~171K evidence-first examples as Tiny 30M round 3, all generated with Qwen3-8B (Apache-2.0) over FineWeb pages and checked.
• Export: SmoothQuant (an exact weight rewrite that moves activation outliers into the weights), then int8 dynamic quantisation with per-channel scales. Plain per-tensor int8 lost most of the site answers; this version matches fp32 on the site questions.

Results

(PyTorch, threshold 0.3. The int8 file on the site questions: 10 right / 8 wrong shown.)

Speed (onnxruntime CPU, 1 thread, int8): ~0.3 s for a 512-token prompt and ~3.4 ms per token.

Experimental: ant-10b/

The same architecture continued for 7.0B more pretraining tokens (9.9B in total; held-out loss 3.33), then the same fine-tune and export. On held-out pages it is better (answer F1 0.33 / 0.32, refusal AUROC 0.988, source picked among distractors 0.83), but on the portfolio's own 24 questions it is worse than the main model (4 right / 14 wrong shown vs 9 / 9): it answers more often where the main model shows sources, and some of those answers come from the wrong passage. Same inputs, outputs and decoding as the main file.

Limitations

This is an experiment, not a production assistant. On a real personal site it answers well under half of factual questions correctly, and it can quote a real but irrelevant sentence and answer from it, which a quote check cannot catch. Use it with the sources shown, and don't rely on it for facts that matter.

Licence and attribution

Apache-2.0. Pretraining data: FineWeb and FineWeb-Edu by Hugging Face, made available under the ODC-By 1.0 licence (and subject to Common Crawl's terms of use). Fine-tuning data was generated with Qwen3-8B (Apache-2.0) over FineWeb pages.