Peregrine-V1

Peregrine-V1 is a 48.4M-parameter English base language model trained from scratch for 39.5 billion tokens. It is designed for text completion and continuation scoring.

Model series

Peregrine-V1 is the larger sibling of Hummingbird-V2 in the same small language model project. It reuses the Hummingbird-V2 data mix with a new 16K tokenizer, and was trained from a fresh initialization.

Architecture

The configured maximum context is not a claim of validated long-context quality.

Training and data

The model was trained on this mix:

Validation and held-out splits of 25M tokens each are separate from training. Training used Muon and AdamW, CUDA BF16, seed 42, a 512-token context, and 196,608 tokens per optimizer update, with a warmup-stable-decay schedule: a constant 3e-4 learning rate to 34B tokens, then decay. The released checkpoint is at 39.5B tokens, inside the decay. See training data for source details and training provenance for the full recipe.

Zero-shot evaluation

Author-run zero-shot results for the 39.5B-token checkpoint. Task scores are normalized continuation accuracy in percent; the Intelligence Index is a chance-normalized composite.

These evaluations selected the released checkpoint as the best of 74 evaluated checkpoints, so the headline score is optimistic. The mean index of the last four checkpoints (38.5B to 40B tokens) is 17.46, and the index has an item-sampling standard error of about 0.8 points. Results have not been independently verified.

Use

Load juinron/Peregrine-V1 with Transformers and the packaged custom code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "juinron/Peregrine-V1"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

inputs = tokenizer("The color of the sky is", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

trust_remote_code=True loads the packaged model code. Review it before use.

Limitations

Peregrine-V1 is small and English-focused. It can make factual or reasoning errors and is not instruction-tuned or safety-aligned. Its arithmetic is near chance outside a few question types. Do not rely on it for consequential decisions.

License

The model package is released under Apache-2.0. Third-party dataset notices are included in NOTICE and TRAINING_DATA.md.