Vertex Pico v1

Tiny • 0.8M Parameters • Base model

Vertex Pico v1 is a 0.8M-parameter base language model (798,726 parameters) trained from scratch by Vertex Research on about 155M tokens. It is the first release in the Vertex Pico series, a family of tiny models built to get the most out of the smallest sizes, with better versions already in training.

Training

• About 155M tokens, one epoch.
• Data: ClimbMix, DCLM-Edu, FineWeb-Edu, UltraX, TxT360 QA, Wikipedia, WikiHow, FineMath, MegaMath QA, SciQ. ClimbMix is licensed CC-BY-NC 4.0.
• 3 layers, width 128, 2 heads, tied embeddings, RoPE; byte-level BPE tokenizer with 3,072 tokens.
• Context: 1,024-token attention window (sliding window beyond that), up to 2,048 positions.

Benchmarks

Zero-shot, float32, scored on every item of each evaluation set (no subsampling): HellaSwag 10,042 (validation), ARC-Easy 2,376 (test), ARC-Challenge 1,172 (test), PIQA 1,838 (validation), ArithMark-3 1,000. Best score in each column in bold.

_{HellaSwag, ARC and PIQA: lm-evaluation-harness, acc_norm. ArithMark-3: the official ArithMark-3 script. Intelligence Index: the Open SLM Leaderboard formula. Other models: scores as listed on the Open SLM Leaderboard. Vertex Pico v1: our own evaluation, pending leaderboard verification.}

Usage

The model uses custom code, so trust_remote_code=True is required.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VertexResearch/Vertex-Pico-v1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-Pico-v1", trust_remote_code=True)

ids = tok("Children learn best when", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=15, do_sample=False, repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

This is a tiny base model for research and benchmarking. It is not instruction-tuned, and its text is often incoherent and factually wrong. Do not use it for anything that matters.