HobbyLM-Chat (500M MoE, instruction-tuned)

HobbyLM-Chat is the instruction-tuned conversational model — HobbyLM-Base taken through SmolTalk supervised fine-tuning and a SmolLM2-style UltraFeedback DPO pass. The jump from base is large: it holds a coherent persona, follows instructions, and (with a repetition penalty) produces varied, flowing prose.

It's part of the HobbyLM family — a 500M sparse-MoE model (and its variants) built from scratch on a hobby budget: FineWeb, a handful of Modal H100 hours, a lot of ablations, and a from-scratch Rust engine (hobby-rs) to run it on a laptop CPU.

Intended use

General single- and multi-turn chat / instruction following. Prompt it with the trained SYSTEM: / USER: / ASSISTANT: turn format, and decode with a repetition penalty ≈1.3 (this is what tames the small-model repetition tendency).

Architecture

Every HobbyLM variant shares one core: a sparse Mixture-of-Experts (MoE) decoder in the modern small-MoE style (DeepSeek-V3 / OLMoE lineage), where each design choice was picked by ablation rather than by guesswork.

The full ablation log (QK-norm is the single biggest lever; aux-loss-free beats classic aux-loss; ≥32 experts and top-6 help; embedding-scaling hurt) lives in the project's architecture notes.

Benchmarks

0-shot multiple-choice, our harness. Note that MC benchmarks measure knowledge, not chat quality — the goal of this checkpoint is conversational fluency, which these tasks don't capture. The small dip vs the base model is the usual alignment tax.

Reasoning tasks (ARC, WinoGrande) held or improved; BoolQ dropped the most — chat phrasing fits the log-likelihood format worse, not a capability loss. This is healthy for a ~500M chat model (SmolLM-360M range).

How these were measured. All language-model scores are 0-shot through our own port of EleutherAI's lm-evaluation-harness (a custom MoELMWrapper that runs log-likelihood scoring over the HobbyLM MoE + GPT-2 tokenizer). Reference models in the comparison table were run through the identical harness and task set, so the numbers are apples-to-apples with ours — they are not copied from other model cards. We validated the harness against published cards (e.g. TinyLlama 52.75 vs card 52.99). These are small research models: read the numbers in context, not as leaderboard claims.

Usage

Python (PyTorch reference implementation)

HobbyLM is a custom sparse-MoE architecture — there's no transformers AutoModel for it, so load it with the small reference implementation from the GitHub repo:

# HobbyLM is a CUSTOM sparse-MoE architecture, so load it with the reference implementation —
# NOT transformers.AutoModelForCausalLM (there is no AutoModel mapping for this arch).
# pip install torch safetensors tiktoken huggingface_hub
# git clone https://github.com/harishsg993010/HobbyLM && cd HobbyLM

import json, torch, tiktoken
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from hobbylm.config import ModelConfig
from hobbylm.model import MoETransformer
from hobbylm.generate import generate

repo = "rootxhacker/HobbyLM-Chat"
cfg = ModelConfig(**{k: v for k, v in json.load(open(hf_hub_download(repo, "config.json"))).items() if k != "preset"})
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
cfg.expert_backend = "grouped" if device.type == "cuda" else "bmm"

model = MoETransformer(cfg).to(device).eval()
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))

enc = tiktoken.get_encoding("gpt2")
prompt = "USER: Give me three tips for better sleep.\nASSISTANT:"
ids = torch.tensor([enc.encode_ordinary(prompt)], device=device)
out = generate(model, ids, max_new_tokens=64, temperature=0.7, top_k=0, device=device,
               repetition_penalty=1.3)               # temperature=0.0 for greedy
print(enc.decode(out[0].tolist()))

Prompt it with the trained USER: / ASSISTANT: turn format (a leading SYSTEM: turn is optional). A repetition penalty around 1.3 is recommended.

GGUF + hobby-rs (CPU)

GGUF builds (architecture hobbylm) live in rootxhacker/HobbyLM-gguf. They load directly in the from-scratch hobby-rs CPU engine — stock llama.cpp won't load them without registering the hobbylm architecture first.

hobby-rs --model HobbyLM-Chat.gguf --prompt "..." --n 64

Training

SFT on ~1.5M chat trajectories (smol-smoltalk + the conversational smoltalk2 subsets), loss on assistant turns only; then UltraFeedback DPO (β=0.1) — the SmolLM2 recipe. SFT loss → ~1.50, DPO preference accuracy 0.50 → 0.64.

Limitations

• Carries the 500M ceiling: factual hallucination, and weak adherence to strict output formats (e.g. exact syllable counts).
• Use a repetition penalty at decode time; greedy decoding can loop.
• Not safety-aligned — no RLHF safety tuning.

License

Apache-2.0. Weights aren't a substitute for judgement — this is a research / hobby model at the 500M scale, not a production system.