language:
Ember-2 is a state-of-the-art, ultra-compact language model developed by SurjoLabs. It ranks #1 on the Open SLM Leaderboard for sub-3M parameter models with an Intelligence Index of 7.21, proving that recurrent weight-sharing and value-subtraction attention (XSA) can achieve exceptional general and mathematical reasoning at micro scales.
Ember-2 is built to demonstrate that specialized recurrent architectures can dramatically outperform standard transformers under extreme parameter constraints (under 3M parameters).
Evaluated at 0-shot using normalized accuracy (acc_norm):
| Benchmark | Ember (v1) | Ember-2 |
|---|---|---|
| HellaSwag | 27.35% | 27.28% |
| ARC-Easy | 33.25% | 33.42% |
| ARC-Challenge | 21.08% | 22.01% |
| PIQA | 54.41% | 55.11% |
| ArithMark-3 | 32.90% | 35.90% |
| Intelligence Index | 5.94 | 7.21 (#1) |
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SurjoLabs/Ember-2"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16
).cuda()
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
We would like to thank AxiomicLabs for pioneer work proving the token-efficiency of the XSA attention mechanism.
Ember-2 is an experimental sub-3M parameter research model developed under the broader Surjo Project. While it demonstrates strong reasoning for its size, output fluency is naturally constrained by parameter capacity. The custom modeling implementation is intended for research and benchmark evaluation.
checkpoint-1000/model.safetensors 12 MBcheckpoint-10000/model.safetensors 12 MBcheckpoint-10500/model.safetensors 12 MBcheckpoint-11000/model.safetensors 12 MBcheckpoint-11500/model.safetensors 12 MBcheckpoint-12000/model.safetensors 12 MBcheckpoint-12500/model.safetensors 12 MBcheckpoint-13000/model.safetensors 12 MBcheckpoint-13500/model.safetensors 12 MBcheckpoint-14000/model.safetensors 12 MBcheckpoint-14500/model.safetensors 12 MBcheckpoint-1500/model.safetensors 12 MBcheckpoint-15000/model.safetensors 12 MBcheckpoint-15500/model.safetensors 12 MBcheckpoint-16000/model.safetensors 12 MBcheckpoint-16500/model.safetensors 12 MBcheckpoint-17000/model.safetensors 12 MBcheckpoint-17500/model.safetensors 12 MBcheckpoint-18000/model.safetensors 12 MBcheckpoint-18500/model.safetensors 12 MBcheckpoint-19000/model.safetensors 12 MBcheckpoint-19500/model.safetensors 12 MBcheckpoint-19999/model.safetensors 12 MBcheckpoint-2000/model.safetensors 12 MBcheckpoint-2500/model.safetensors 12 MBcheckpoint-3000/model.safetensors 12 MBcheckpoint-3500/model.safetensors 12 MBcheckpoint-4000/model.safetensors 12 MBcheckpoint-4500/model.safetensors 12 MBcheckpoint-500/model.safetensors 12 MBcheckpoint-5000/model.safetensors 12 MBcheckpoint-5500/model.safetensors 12 MBcheckpoint-6000/model.safetensors 12 MBcheckpoint-6500/model.safetensors 12 MBcheckpoint-7000/model.safetensors 12 MBcheckpoint-7500/model.safetensors 12 MBcheckpoint-8000/model.safetensors 12 MBcheckpoint-8500/model.safetensors 12 MBcheckpoint-9000/model.safetensors 12 MBcheckpoint-9500/model.safetensors 12 MBmodel.safetensors 12 MBcheckpoint-1000/optimizer.bin 9 MBcheckpoint-10000/optimizer.bin 9 MBcheckpoint-10500/optimizer.bin 9 MBcheckpoint-11000/optimizer.bin 9 MBcheckpoint-11500/optimizer.bin 9 MBcheckpoint-12000/optimizer.bin 9 MBcheckpoint-12500/optimizer.bin 9 MBcheckpoint-13000/optimizer.bin 9 MBcheckpoint-13500/optimizer.bin 9 MBcheckpoint-14000/optimizer.bin 9 MBcheckpoint-14500/optimizer.bin 9 MBcheckpoint-1500/optimizer.bin 9 MBcheckpoint-15000/optimizer.bin 9 MBcheckpoint-15500/optimizer.bin 9 MBcheckpoint-16000/optimizer.bin 9 MBcheckpoint-16500/optimizer.bin 9 MBcheckpoint-17000/optimizer.bin 9 MBcheckpoint-17500/optimizer.bin 9 MBcheckpoint-18000/optimizer.bin 9 MBcheckpoint-18500/optimizer.bin 9 MBcheckpoint-19000/optimizer.bin 9 MBcheckpoint-19500/optimizer.bin 9 MBcheckpoint-19999/optimizer.bin 9 MBcheckpoint-2000/optimizer.bin 9 MBcheckpoint-2500/optimizer.bin 9 MBcheckpoint-3000/optimizer.bin 9 MBcheckpoint-3500/optimizer.bin 9 MBcheckpoint-4000/optimizer.bin 9 MBcheckpoint-4500/optimizer.bin 9 MBcheckpoint-500/optimizer.bin 9 MBcheckpoint-5000/optimizer.bin 9 MBcheckpoint-5500/optimizer.bin 9 MBcheckpoint-6000/optimizer.bin 9 MBcheckpoint-6500/optimizer.bin 9 MBcheckpoint-7000/optimizer.bin 9 MBcheckpoint-7500/optimizer.bin 9 MBcheckpoint-8000/optimizer.bin 9 MBcheckpoint-8500/optimizer.bin 9 MBcheckpoint-9000/optimizer.bin 9 MBcheckpoint-9500/optimizer.bin 9 MBoptimizer.bin 9 MBcheckpoint-1000/optimizer_1.bin 5 MBcheckpoint-10000/optimizer_1.bin 5 MBcheckpoint-10500/optimizer_1.bin 5 MBcheckpoint-11000/optimizer_1.bin 5 MBcheckpoint-11500/optimizer_1.bin 5 MBcheckpoint-12000/optimizer_1.bin 5 MBcheckpoint-12500/optimizer_1.bin 5 MBcheckpoint-13000/optimizer_1.bin 5 MBcheckpoint-13500/optimizer_1.bin 5 MBcheckpoint-14000/optimizer_1.bin 5 MBcheckpoint-14500/optimizer_1.bin 5 MBcheckpoint-1500/optimizer_1.bin 5 MBcheckpoint-15000/optimizer_1.bin 5 MBcheckpoint-15500/optimizer_1.bin 5 MBcheckpoint-16000/optimizer_1.bin 5 MBcheckpoint-16500/optimizer_1.bin 5 MBcheckpoint-17000/optimizer_1.bin 5 MBcheckpoint-17500/optimizer_1.bin 5 MBcheckpoint-18000/optimizer_1.bin 5 MBcheckpoint-18500/optimizer_1.bin 5 MBcheckpoint-19000/optimizer_1.bin 5 MBcheckpoint-19500/optimizer_1.bin 5 MBcheckpoint-19999/optimizer_1.bin 5 MBcheckpoint-2000/optimizer_1.bin 5 MBcheckpoint-2500/optimizer_1.bin 5 MBcheckpoint-3000/optimizer_1.bin 5 MBcheckpoint-3500/optimizer_1.bin 5 MBcheckpoint-4000/optimizer_1.bin 5 MBcheckpoint-4500/optimizer_1.bin 5 MBcheckpoint-500/optimizer_1.bin 5 MBcheckpoint-5000/optimizer_1.bin 5 MBcheckpoint-5500/optimizer_1.bin 5 MBcheckpoint-6000/optimizer_1.bin 5 MBcheckpoint-6500/optimizer_1.bin 5 MBcheckpoint-7000/optimizer_1.bin 5 MBcheckpoint-7500/optimizer_1.bin 5 MBcheckpoint-8000/optimizer_1.bin 5 MBcheckpoint-8500/optimizer_1.bin 5 MBcheckpoint-9000/optimizer_1.bin 5 MBcheckpoint-9500/optimizer_1.bin 5 MBoptimizer_1.bin 5 MBHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo SurjoLabs/Ember-2 ./model-folder