🐟 Fish LM: 68M Dense Small Language Model (SFT)

Govind222/68m-base-guppy-sft is a high-efficiency 68.96M parameter Dense Small Language Model (SLM) designed for ultra-fast edge inference, local device hosting, and conversational dialog.

Built from scratch using transformer-toolkit, the model was pretrained on 4 Billion tokens of FineWeb and instruction fine-tuned on 57,000 curated dialog examples (arman-bd/guppylm-60k-generic).

Live Interactive Demo: https://fish-lm.runlog.in/ (hosted entirely on an edge Android smartphone)
Training Notebook: Fish_LM.ipynb on GitHub
Live Runlog Telemetry: solar-aurora-298 on Runlog

⚡ Highlights & Key Metrics

Pure Dense Transformer: 100% dense computation with zero MoE overhead, maximizing per-token compute density and caching efficiency on edge CPU/GPU.
Modern Architecture: Grouped-Query Attention (GQA), SwiGLU feed-forward layers, RoPE rotary positional embeddings, and pre-layer RMSNorm.
Superb Perplexity: Achieves 1.275 SFT validation perplexity (val_loss: 0.2429) on dialog tasks.
Extreme Efficiency: Full pretrain + SFT executed for under ~$14 on a single NVIDIA L40S GPU in ~14 hours.
Mobile-Optimized: Serves real-time tokens at low latency (~35ms TTFT) on mobile ARM big.LITTLE CPUs.

📊 Evaluation & Benchmarks

Zero-shot evaluations performed via lm-evaluation-harness demonstrate that Fish LM 68M remains highly competitive with models nearly 2× its size:

Benchmark	Metric	Fish LM (68M Dense)	Pythia 70M	Pythia 160M	GPT-Neo 125M	OPT 125M
PIQA	Accuracy	58.1%	58.6%	61.4%	62.3%	61.8%
Winogrande	Accuracy	48.7%	51.6%	50.4%	49.5%	51.5%
ARC-Easy	Acc Norm	31.4%	34.2%	—	38.9%	39.8%
Overall Average	Mean Acc	39.6%	41.4%	43.7%	43.5%	43.9%

⚙️ Model Architecture

Parameter	Value	Specification Notes
Total Parameters	`68,957,056` (~68.96M)	Exact non-embedding + embedding tensor weight sum
Architecture Type	Dense Decoder-Only SLM	Pure dense transformer (no MoE routing)
Layers (`n_layers`)	`12`	Uniform transformer blocks
Hidden Dim (`dim`)	`512`	Model dimension ($d_{\text{model}}$)
Query Heads (`n_heads`)	`8`	Head dimension = 64
KV Heads (`n_kv_heads`)	`2`	4:1 GQA ratio for compressed KV cache memory
FFN Intermediate Dim	`1536`	$3 \times d_{\text{model}}$
FFN Activation	`swiglu`	Dense SwiGLU gating (`w1`, `w2`, `w3`)
Positional Encoding	`rope`	Rotary Position Embeddings
Normalization	`rmsnorm`	Pre-norm with $\epsilon = 1\text{e-}6$
Vocabulary Size	`32000`	Custom Rust BPE Tokenizer
Max Sequence Length	`512`	Context window length
Weight Tying	`false`	Untied input embeddings and output LM head

📈 Training Details

1. Pretraining

Base Checkpoint: Govind222/68m-base
Pretraining Corpus: 4.0 Billion tokens from Hugging Face's FineWeb (sample-10BT)
Objective: Standard autoregressive next-token prediction with cosine decay schedule

2. Supervised Fine-Tuning (SFT)

Dataset: 57,000 instruction & conversation pairs from arman-bd/guppylm-60k-generic
Training Framework: transformer-toolkit
Telemetry & Tracking: Live monitored on Runlog (Run: solar-aurora-298)
Hardware: 1× NVIDIA L40S GPU
Training Throughput: ~6,064 tokens/sec
Final SFT Validation Loss: 0.2429
Final SFT Perplexity: 1.275

🚀 Quickstart & Inference

Prerequisites

pip install torch
pip install git+https://github.com/Barbade22/transformer-toolkit.git

Loading and Generating Text

import torch
from transformer_toolkit import TransformerConfig, Transformer, RustBPETokenizer

# 1. Initialize Architecture
config = TransformerConfig(
    vocab_size=32000,
    dim=512,
    n_layers=12,
    n_heads=8,
    n_kv_heads=2,
    ffn="swiglu",
    hidden_dim=1536,
    norm="rmsnorm",
    pos_enc="rope",
    dropout=0.0,
    tie_weights=False,
    max_seq=512,
    use_kv_cache=True,
)

model = Transformer(config)

# 2. Load Weights & Tokenizer
state_dict = torch.load("model.pt", map_location="cpu")
if isinstance(state_dict, dict) and "model" in state_dict:
    state_dict = state_dict["model"]
model.load_state_dict(state_dict)
model.eval()

tokenizer = RustBPETokenizer()
tokenizer.load("tokenizer.json")

# 3. Format Input & Generate
prompt = "<|im_start|>user\nHello! Who are you?<|im_end|>\n<|im_start|>assistant\n"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

output_ids = model.generate(
    input_ids,
    max_new=50,
    temperature=0.4,
    top_k=50,
)

response = tokenizer.decode(output_ids[0].tolist())
print(response)

📱 Edge & On-Device Deployment

Fish LM was specifically benchmarked and optimized for constrained edge hardware:

Host Architecture: Android / Termux on ARMv8/ARMv9 big.LITTLE core topology
Thread Scheduling: Peak throughput achieved on 2 Big CPU cores (OMP_NUM_THREADS=2), preventing thermal throttling and Little-core barrier latency.
Memory Footprint: Less than ~280 MB RAM at runtime with automatic idle watchdog unloading.

📚 Acknowledgments & Citations

Built and trained with transformer-toolkit by @Barbade22.
Dialog dataset provided by GuppyLM by Arman (@arman-bd).
Experiment tracking and live dashboard provided by Runlog.