Gemma 3 110M English Only
This is Gemma 3 270M with the pika 3 tokenizer used and with retrained embedding and output layers, afterwards a gentle retrain.
Chat template
Note: This uses the Zephyr chat template.
Training
It was trained on these using 1 epoch, sparse adam optimizer, and 32 batch size:
- HuggingFaceTB/cosmopedia-20k
- HuggingFaceTB/smol-smoltalk (first 23,000 rows)
- HuggingFaceTB/everyday-conversations-llama3.1-2k
Limitations
The quality of this model needs to be better tested. It will also poorly handle non-English languages.