Prochlo-190M-SFT
Supervised fine-tuned version of Prochlo-190M-Base — a 190.5M-parameter dense English SLM trained from scratch.
Model family
| Repo | Stage |
|---|---|
CyanoAI/Prochlo-190M-Base |
Pre-trained base |
CyanoAI/Prochlo-190M-SFT |
Supervised fine-tune (this repo) |
CyanoAI/Prochlo-190M |
Preference-aligned (DPO + RLAIF) — in training |
All variants share architecture, tokenizer, and vocabulary.
Training
| Base | CyanoAI/Prochlo-190M-Base (~7.7B pre-training tokens, 12 layers, vocab 50,261) |
| Data | HuggingFaceTB/smol-smoltalk, ~460k English conversations, chat-formatted with <|user|> / <|assistant|> |
| Method | Full-parameter SFT, 2 epochs, lr 3e-4 cosine, batch 216, max_len 1024, fp16, loss masked to assistant tokens |
| Steps | 96,135 |
| Final loss | 1.9308 (train), ~1.94 (eval) |
| Duration | 27 hours |
Evaluation
26-item behavioral rubric (format adherence, anti-sycophancy, code, concision): 61.5% pass.
Strong on code instructions and factual QA. Weak on concise replies, long multi-turn coherence, and arithmetic. English only — Chinese input produces garbage by design.
Usage
The chat format is plain text with control tokens, not a chat template:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("CyanoAI/Prochlo-190M-SFT")
model = AutoModelForCausalLM.from_pretrained("CyanoAI/Prochlo-190M-SFT", dtype="float16").cuda()
prompt = "<|user|>Explain what a closure is.<|end|><|assistant|>"
ids = tok(prompt, return_tensors="pt").input_ids.cuda()
out = model.generate(ids, max_new_tokens=256, do_sample=False,
eos_token_id=tok.convert_tokens_to_ids("<|end|>"))
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
Note: weights are sharded across 32 safetensors files with a model.safetensors.index.json; from_pretrained handles this transparently.
License
Apache-2.0