Ethoric-10M

Model Summary

Ethoric-10M is a decoder-only small language model with 9.97M parameters. It is built on a modern transformer foundation of RoPE, grouped-query attention and SwiGLU, and adds four changes on top. Peri-LN normalization applies RMSNorm both before and after every sublayer to keep training stable. Canon layers are short causal depthwise convolutions that mix each token with its 3 previous tokens before attention and before the MLP. Per-head QK-norm normalizes the queries and keys in each attention head, and a headwise sigmoid gate lets each head control how much of its output passes through. The model was trained on 15B tokens, followed by a 4B-token cooldown.

Model Architecture

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "techdotus/Ethoric-10M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

ids = tok("The climate change", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=32, do_sample=False,
                         eos_token_id=1)  # greedy
print(tok.decode(out[0], skip_special_tokens=True))

The model's custom generate() supports greedy decoding and top-k sampling (do_sample=True, temperature, top_k). Other options such as repetition_penalty are ignored. Prompt plus generated tokens must stay within 512 tokens.

Benchmark Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology.

Comparison Results

All scores are 0-shot acc_norm, using the Open SLM Leaderboard methodology. Scores for the other models are from the Open SLM Leaderboard. Bold marks the best result in each column.

Training Details

Ethoric-10M was trained in two stages.

Training packs multiple documents into each 512-token sequence. Document masking keeps them separate: attention and Canon convolutions never reach across an EOS token.

Training Data

The cooldown added new math sources such as FineMath, more question-answer and tutorial-style text such as FinePhrase, ELI5 and GooAQ, and dropped Tiny Strange Textbooks, High-Quality English Sentences and ScienceQA.

† ELI5, GooAQ, FactBase-DCLM, Cosmopedia-v2, Natural Questions, Pretrain-Behaviors, UltraData-Math

Limitations

• Tiny model: with 9.97M parameters and 256-dim hidden states, generations are often repetitive, incoherent or factually wrong. The model is a research artifact for studying small-scale LMs, not an assistant.
• Short context: 512 tokens maximum. RoPE tables are not built beyond that length, and the small 5,048-token vocabulary means 512 tokens covers less text than in models with larger vocabularies.
• English only: trained on English web, educational, synthetic, question-answer and math text.
• Not instruction-tuned or safety-aligned: it may reproduce biases present in the data.
• Benchmark scores are near chance on some tasks: ARC-Challenge (24.66%) and HellaSwag (28.75%) are close to the 25% random baseline, so small differences between models on these tasks should be read with caution.

License

MIT

Citation

@misc{ethoric10m2026,
  title        = {{Ethoric-10M}: A Small Language Model with Stabilized,
                  Gated Attention},
  author       = {{Tech.us Team}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/techdotus/Ethoric-10M}}
}