TR-HASH MoE 200M — 160B-source Full SFT v2
Full-parameter instruction SFT of
AETHORIA-AI/TR-HASH-MoE-200M-160B-Refinement on the audited
AETHORIA-AI/TR-HASH-MoE-200M-SFT-v2-300K. This release is
not LoRA or QLoRA: all 201.2M model parameters were trainable.
The root model.safetensors is the epoch 3 / step
5,982 checkpoint, selected for the strongest full-PIQA result
and lowest held-out SFT loss across the three epochs.
Tokenizer compatibility notice (23 August 2026): this released SFT remains a 32,000-token checkpoint and must be loaded with the tokenizer files stored in this repository. It was trained from the pre-migration 32,000-token Refinement revision. The canonical Base and Refinement roots now use 32,004 tokens; a replacement SFT must be retokenized from raw text and explicitly train IDs 32,000–32,003. Do not pair this checkpoint with the standalone 32,004-token tokenizer or resize it as a substitute for retraining.
Results
| Epoch | Step | Held-out SFT loss | SFT ppl | PIQA acc | PIQA acc_norm |
|---|---|---|---|---|---|
| 1 | 1,994 | 0.990943 | 2.69 | 67.90% | 68.93% |
| 2 | 3,988 | 0.963912 | 2.62 | 67.85% | 68.82% |
| 3 | 5,982 | 0.959617 | 2.61 | 68.01% | 69.10% |
Released checkpoint benchmark panel
| Benchmark | Split | Examples | Accuracy | Accuracy (length-normalized) | Evaluation backend |
|---|---|---|---|---|---|
| PIQA | validation | 1,838 | 68.01% | 69.10% | PyTorch FP16, custom Triton |
| ARC-Easy | test | 2,376 | 57.24% | 52.86% | MLX FP16 |
| ARC-Challenge | test | 1,172 | 27.13% | 29.01% | MLX FP16 |
| ARC Combined (micro) | test | 3,548 | 47.29% | 44.98% | Derived from both ARC test splits |
| HellaSwag | validation | 10,042 | 33.21% | 38.74% | MLX FP16 |
All benchmark evaluations use zero-shot causal continuation log-likelihood, no
chat template and a maximum sequence length of 2,048. Accuracy selects the
choice with the highest total continuation log-likelihood; the normalized
metric selects by mean continuation log-likelihood per scored token. PIQA was
evaluated from the native epoch-3 checkpoint. ARC and HellaSwag were
evaluated from an FP16 MLX conversion of the same root F32 SafeTensors weights.
ARC Combined is the micro-average over all 3,548 ARC-Easy and ARC-Challenge
test examples, not the arithmetic mean of the two percentages.
Machine-readable reports are published under reports/sft-v2-300k/.
Compact-model comparison
| Model | Parameters | PIQA acc | ARC-Easy acc | ARC-Challenge acc | Combined ARC acc | HellaSwag acc |
|---|---|---|---|---|---|---|
| TR-HASH MoE 200M Full SFT | 201.2M | 68.01% | 57.24% | 27.13% | 47.29% | 33.21% |
| GPT-2 Large | 774M | — | 53.11% | 21.76% | 42.76% | — |
| Pythia-410M | 410M | — | 52.02% | 21.42% | 41.91% | — |
| GPT-2 Medium | 355M | — | 49.16% | 21.67% | 40.08% | — |
| OPT-350M | 350M | — | 43.98% | 20.82% | 36.33% | — |
| GPT-2 Small | 124M | 62.89% | 43.81% | 19.03% | ≈35.63% | 28.92% |
| OPT-125M | 125M | 63.00% | 43.60% | 19.10% | ≈35.51% | 29.20% |
| Pythia-160M | 160M | 62.73% | 43.52% | 18.77% | 35.34% | — |
Combined ARC is weighted by the public test-set sizes (2,376 ARC-Easy and 1,172 ARC-Challenge examples). GPT-2 Small and OPT reference scores come from the AMD-LLM lm-evaluation-harness comparison; their combined values are approximate because the published component scores are rounded. Pythia-160M uses EleutherAI's official zero-shot result. GPT-2 Medium, GPT-2 Large and Pythia-410M were evaluated locally in MLX FP16 with the same causal-choice evaluator as TR-HASH. OPT-350M used the identical prompt and scoring formula in PyTorch MPS FP16 because MLX does not implement the OPT architecture. The 124M–160M references were reported through lm-evaluation-harness, so that part of the comparison is informative rather than a claim of bit-identical evaluation runtimes.
Training recipe
| Setting | Value |
|---|---|
| Method | Full-parameter supervised fine-tuning |
| Source | Refinement step 8,156 (about 162B prior token exposures) |
| Dataset | 300,000 train / 3,000 held-out examples |
| Tokenized corpus | 202,948,693 train tokens; no truncation |
| Supervision | Final assistant turn only; prior assistant turns masked |
| Epochs | 3 |
| Context | 2,048 tokens |
| Tokenizer | TR-HASH 32,000-token vocabulary; EOS </s> (ID 0) |
| Optimizer | AdamW, betas 0.9 / 0.95, weight decay 0.1 |
| LR | 2e-5 peak, 3% warmup, continuous cosine decay |
| Precision | BF16 training |
| Root SafeTensors precision | float32 |
| Kernels | Liger required; custom Triton enabled |
Architecture and loading
201.2M parameters, 16 decoder layers, GQA (14 query heads / 2 KV heads), four stored deterministic token-ID-routed experts with top-2 activation, an always-on shared SwiGLU path and tied embeddings. The persisted multi-hash routing tables are part of the checkpoint.
The repository includes an autonomous Transformers adapter. Load it with:
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
License
The released SFT checkpoint is Apache-2.0. Source datasets retain their own licenses and terms; see the dataset manifest for the per-source audit.