A modern, highly efficient 108 Million parameter (0.1B) Thai-English Small Language Model (SLM) trained on over 504 Million tokens across 6,500 training steps.
Developed by Jommarn, this model incorporates cutting-edge architectures inspired by Gemma 2 and DeepSeek-V3 to maximize reasoning and linguistic capability under a strict compact footprint.
| Feature | Specification | Rationale |
|---|---|---|
| Parameters | 108,160,512 (~108M) | Extremely lightweight, runs on mobile & edge devices |
| Layers | 14 Transformer Blocks | Deeper network for strong semantic abstraction |
| Attention | Interleaved Local-Global | 7 Layers Sliding Window (512) + 7 Layers Global Attention (Gemma 2 style, saves 50% KV cache) |
| GQA | 6 Query Heads, 2 KV Heads | Grouped Query Attention for fast, memory-efficient decoding |
| MTP | Multi-Token Prediction | Dual-loss prediction ($T+1$ and $T+2$) inspired by DeepSeek-V3 |
| Activations | SwiGLU & RMSNorm | Superior gradient propagation and numerical stability |
| Embeddings | Tied Input-Output (16K) | Weight sharing saves 12.5M parameters |
| Context Length | 1,024 tokens (Window 512) | Optimized for compact dialogues and reasoning |
WangchanLION-Web), Thai food, and tech news. Total accumulated tokens reached >406M tokens.L(T+1) reaching 0.90 - 1.05!Use the standalone PyTorch modeling code modeling_0_1b.py with Hugging Face weights:
import torch
from transformers import PreTrainedTokenizerFast
from huggingface_hub import hf_hub_download
import sys
# 1. Load Tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained("Jommarn/thai-0.1b-tokenizer")
# 2. Download Model Checkpoint
ckpt_path = hf_hub_download(repo_id="Jommarn/thai-0.1b-slm", filename="midtrain_step_6500/model_checkpoint.pt")
checkpoint = torch.load(ckpt_path, map_location="cpu")
# 3. Initialize Model Architecture
from modeling_0_1b import Thai0_1BForCausalLM
model = Thai0_1BForCausalLM(checkpoint["config"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
# 4. Generate
prompt = "กรุงเทพมหานครเป็นเมืองหลวงและศูนย์กลางทางเศรษฐกิจของประเทศไทย โดยมี"
input_ids = tokenizer.encode(prompt, return_tensors="pt")
with torch.no_grad():
output = model.generate(input_ids, max_new_tokens=150, temperature=0.7, top_k=30)
print(tokenizer.decode(output[0].tolist()))
Apache-2.0. Free for open research and commercial use.
model_checkpoint.pt 1.3 GBmidtrain_step_1250/model_checkpoint.pt 433 MBmidtrain_step_1500/model_checkpoint.pt 433 MBmidtrain_step_1750/model_checkpoint.pt 433 MBmidtrain_step_2000/model_checkpoint.pt 433 MBmidtrain_step_2250/model_checkpoint.pt 433 MBmidtrain_step_250/model_checkpoint.pt 433 MBmidtrain_step_2500/model_checkpoint.pt 433 MBmidtrain_step_2750/model_checkpoint.pt 433 MBmidtrain_step_3000/model_checkpoint.pt 433 MBmidtrain_step_3250/model_checkpoint.pt 433 MBmidtrain_step_3500/model_checkpoint.pt 433 MBmidtrain_step_3750/model_checkpoint.pt 433 MBmidtrain_step_4000/model_checkpoint.pt 433 MBmidtrain_step_4250/model_checkpoint.pt 433 MBmidtrain_step_4500/model_checkpoint.pt 433 MBmidtrain_step_4750/model_checkpoint.pt 433 MBmidtrain_step_500/model_checkpoint.pt 433 MBmidtrain_step_5000/model_checkpoint.pt 433 MBmidtrain_step_5250/model_checkpoint.pt 433 MBmidtrain_step_5500/model_checkpoint.pt 433 MBmidtrain_step_5750/model_checkpoint.pt 433 MBmidtrain_step_6000/model_checkpoint.pt 433 MBmidtrain_step_6250/model_checkpoint.pt 433 MBmidtrain_step_6500/model_checkpoint.pt 433 MBmidtrain_step_6750/model_checkpoint.pt 433 MBmidtrain_step_7000/model_checkpoint.pt 433 MBmidtrain_step_7250/model_checkpoint.pt 433 MBmidtrain_step_750/model_checkpoint.pt 433 MBmidtrain_step_7500/model_checkpoint.pt 433 MBmidtrain_step_7750/model_checkpoint.pt 433 MBmidtrain_step_8000/model_checkpoint.pt 433 MBHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo Jommarn/thai-0.1b-slm ./model-folder