SOMA-S1 is a Somali-first foundation language model (LLM) built to advance natural language processing for Somali. It is a decoder-only, autoregressive Transformer with a sparse Mixture-of-Experts (MoE) feed-forward architecture and Multi-head Latent Attention (MLA). The model was pretrained from scratch on a dedicated Somali-language corpus rather than adapted from an existing pretrained checkpoint.
Somali remains underrepresented in modern AI research and large-scale language-model development. SOMA-S1 is an experimental effort to establish foundational modeling capabilities for the language through a dedicated tokenizer, a Somali-focused training corpus, and a custom neural architecture.
This is a base model, not a conversational assistant. It is trained to continue text rather than follow instructions or answer questions. Generation quality is currently limited, with repetition often emerging after approximately 10–20 generated tokens under greedy decoding. See Generation Settings.
SOMA-S1 uses a sparse Mixture-of-Experts (MoE) Transformer architecture with Multi-head Latent Attention (MLA). Its attention mechanism employs a compressed latent key-value representation, while its feed-forward architecture uses routed experts, activating a subset of the available expert parameters for each token.
The model contains approximately 126.19 million total parameters and 69.57 million active parameters per token, based on the configured top-2 expert routing.
Each Transformer layer contains 18 routed experts and one shared expert. Two routed experts are selected per token, while the shared expert is also used.
The active parameter count includes the shared model parameters and the selected routed experts. The routed expert parameters alone account for approximately 7.08 million active parameters across the 12 layers.
| Component | Configuration |
|---|---|
| Model type | Decoder-only Transformer |
| Training approach | Pretraining from scratch |
| Total parameters | 126.19 million |
| Active parameters per token | 69.57 million |
| Active routed expert parameters | 7.08 million |
| Hidden dimension | 768 |
| Transformer layers | 12 |
| Attention mechanism | Multi-head Latent Attention (MLA) |
| Attention heads | 12 |
| KV latent rank | 128 |
| Q latent rank | 128 |
| QK no-PE head dimension | 32 |
| QK RoPE head dimension | 32 |
| V head dimension | 64 |
| Feed-forward architecture | Mixture of Experts (MoE) |
| Routed experts | 18 per layer |
| Shared experts | 1 per layer |
| Active routed experts | 2 per token |
| MoE expert intermediate size | 128 |
| Vocabulary size | 60,000 |
| Tokenizer | Custom byte-level BPE |
| Maximum context length | 2,048 tokens |
| Normalization | RMSNorm |
| Activation | SiLU |
| Positional encoding | RoPE (θ = 10,000) |
| Tied embeddings | Yes |
Parameter-counting note: Active parameters are calculated by counting all non-routed model parameters, including shared experts, together with the two selected routed experts in each of the 12 layers. The 7.08M figure represents only the selected routed expert weights, not the entire active model.
SOMA-S1 was pretrained from scratch using the Somali NLP Corpus, developed by SOMAST. The dataset brings together Somali-language text from multiple publicly available sources to support natural language processing and language-model research.
While this corpus provides a valuable foundation for Somali language modeling, its coverage, linguistic diversity, data quality, and representation of different types of Somali text require further improvement.
SOMA-S1 currently has limitations in text coherence, factual accuracy, repetitive generation, and long-form text generation. Further development will require broader and higher-quality Somali datasets, more comprehensive data filtering, and additional training and evaluation.
SOMA-S1 should therefore be considered an early-stage research model rather than a mature, general-purpose Somali language model.
The model was trained using a custom data-processing and distributed training pipeline.
This pipeline was designed to support efficient pretraining and make effective use of available GPU resources.
| File | Size | Notes |
|---|---|---|
model.safetensors |
504,852,656 B | FP32 weights |
config.json |
1.2 KB | model_type: deepseek_v2 |
generation_config.json |
234 B | EOS = 3, BOS = 2, PAD = 1 |
tokenizer.json |
4.3 MB | 60,000-token byte-level BPE |
tokenizer_config.json |
320 B | Special tokens and content-only chat template |
LICENSE |
11 KB | Apache-2.0 |
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "boore1/SOMA-S1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.float32
)
prompt = "Soomaaliya waa dal"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(
ids,
max_new_tokens=120,
do_sample=True,
temperature=0.8,
top_p=0.95,
top_k=50,
repetition_penalty=1.2,
pad_token_id=tok.pad_token_id,
)
print(
tok.decode(
out[0][ids.shape[1]:],
skip_special_tokens=True
)
)
The bundled chat template is a content-only passthrough. It concatenates message content without role markers, meaning the model behaves as a raw text-completion model rather than a conversational assistant.
For best results, provide a natural Somali text prefix that the model can continue.
Example:
Soomaaliya waa dal
Recommended generation parameters:
repetition_penalty: 1.2 or higher.temperature: 0.7–0.8.top_p: 0.9–0.95.Greedy decoding can produce repetitive sequences. Sampling and repetition penalties may help reduce repetition, but they do not eliminate the model's underlying limitations.
Future work includes Somali instruction tuning, improved long-context coherence, reduced repetition and hallucination, expanded evaluation, and downstream research in conversational AI, summarization, question answering, and Somali information retrieval.
Apache License 2.0 — see LICENSE.
@misc{soma-s1,
title = {SOMA-S1: A Somali Language Model},
author = {Boore},
year = {2026},
note = {Sparse Mixture-of-Experts Transformer with Multi-head Latent Attention, pretrained from scratch on Somali text},
url = {https://huggingface.co/boore1/SOMA-S1}
}