LiteGPT V2 Base

A 201M-parameter language model built from scratch, with every step on record.

LiteGPT V2 has its own architecture implementation, its own trained tokenizer and its own weights. It was pretrained in 37 hours on a single rented GPU, and every experiment behind it was registered before it ran. Results, charts and run records: github.com/kmkrofficial/LiteGPT.

Highlights

• A big step up from LiteGPT V1: clearly better on five benchmarks and clearly worse on none, and about three times better on code text (bits per byte 1.612 → 0.515).
• Strong code modelling for its size: the lowest code bits per byte of every model under 300M parameters we measured.
• Light to run: 0.59 GiB of GPU memory at batch 8 with LiteGPT's engine, and 1.66 GiB of RAM on CPU.
• Good at structured output once fine-tuned: 91.4% accuracy on Banking77 intents, and valid JSON on 99.99% of MASSIVE requests.
• Loads in Transformers with no remote code.

Results

After fine-tuning on each task:

Every benchmark, including comparisons with 16 other models: results.

How it compares

LiteGPT V2 doesn't reach the top score in these comparisons. What it brings is a model built entirely from scratch, light enough for a laptop, and open down to every run record.

Text-to-data, after fine-tuning each model the same way (mean of 3 seeds, higher is better):

As a base model (higher is better, except bits per byte and memory):

V2's memory figures use its own engine and the others use Transformers.

Model

Use

With Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "kmkrworks/LiteGPT-V2-Base"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
inputs = tokenizer("The capital of France is", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=32)[0], skip_special_tokens=True))

With LiteGPT's own engine, from a clone of the repository:

uv sync --locked
uv run litegpt generate --model LiteGPT-V2-Base "The capital of France is"

Benchmarks were scored through LiteGPT's engine.

Intended use: research, learning, and fine-tuning for your own tasks. It is a base model with no safety tuning, so add your own safeguards before putting it in front of people.

Training data

Pretrained on a mix chosen in registered experiments. Mixture and exact revisions: data_sources.yaml. Attributions: the NOTICE file.

Contains information from FineWeb-Edu, Cosmopedia v2 and FineMath, which are made available under the ODC Attribution License.

Licence

Apache-2.0 for the weights, MIT for the code, with the NOTICE file.

Built with Qwen.