TinyBalls-v1-Base

The base (non-instruct) model: 113.3M parameters, trained from scratch on 2.01B tokens of English text. For the chat and tool-use version see igidn/TinyBalls-110M-V1.

Architecture

Weights are float32 and load as a standard transformers LlamaForCausalLM.

Usage

This is a completion model. There is no chat template, and the ChatML tokens in its vocabulary are untrained, do not prompt it with <|im_start|>.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("igidn/TinyBalls-v1-Base")
model = AutoModelForCausalLM.from_pretrained(
    "igidn/TinyBalls-v1-Base", torch_dtype="float32")

ids = tok("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=48, do_sample=True,
                     temperature=0.8, top_p=0.95)
print(tok.decode(out[0][ids.shape[1]:]))

Documents were packed with <|endoftext|> (id 0) as the separator, which is also the EOS and pad token here. The generation_config.json defaults (temperature=0.8, top_p=0.95, sampling on) are what you want: greedy decoding collapses into repetition on this checkpoint, under greedy sampling the 4-gram repeat rate is 0.94 and distinct-token rate 0.04, versus 0.05–0.09 and 0.38–0.43 under top-p sampling.

Training

Two phases on a 2.01B-token English-only mix (1.5B broad, then a 0.5B quality anneal over the final 25% of steps), one LR schedule spanning both:

Files

License

MIT.

-igidn