VKORE-325M-NPU-Base

The common wisdom was absolute: an inference only NPU cannot train. Not even LoRA fine tuning, let alone pretraining from scratch.

First we proved it wrong at 46M. Now we did it at 325M, seven times larger, on 5 billion tokens of real web text, and matched a B200 GPU.

A 325M parameter base language model pretrained entirely from scratch on a commercial inference only NPU, using backpropagation run on the chip itself. It is the second model of the VKORE NPU training line, after VKORE-46M-NPU.

One line

From 46M to 325M: we grew a language model from zero on a chip built only for inference, and it reached the same evaluation loss as the same model trained on a B200 GPU.

Highlights

• Pretrained from scratch on a commercial inference only NPU (32 chips). No pretrained weights.
• 325M parameters, about 7 times larger than VKORE-46M-NPU.
• 5 billion tokens of real educational web text (FineWeb-Edu), not a toy story set.
• Long, stable training: 76,293 steps over about four days.
• Same quality as GPU: evaluation loss 0.8067 on the NPU vs 0.8070 on a B200 GPU, same model, same data, same schedule.

The journey: 46M to 325M

Going from a short run to tens of thousands of steps at 7 times the size was the main engineering work between the two models. The technique that keeps long training stable on the NPU was developed in house and is under patent application.

NPU vs B200 GPU

The same model was trained twice, from the same initialization, on the same data with the same learning rate schedule: once on 32 NPU chips and once on a B200 GPU. Both were evaluated with the same script on the same 256 fixed evaluation windows (65,536 byte tokens) taken from the training corpus, and again on 1,024 windows of FineWeb-Edu text that was never used for training (sample/10BT shard 013; the training corpus was built from shards 000 and 001 only). Both models are published here, so the comparison can be reproduced with evaluate_unseen.py. The B200 weights are in b200_reference/.

On the unseen text, the paired difference over identical windows is -0.0001 (NPU minus GPU) with a standard error of 0.0007, and the NPU model is lower on 50.1% of windows.

The differences (0.0003 and 0.0001) are far inside measurement noise. The NPU reached GPU quality.

The B200 GPU weights are included in this repository under b200_reference/, so anyone can compare the two runs directly:

from transformers import AutoModelForCausalLM
npu = AutoModelForCausalLM.from_pretrained("FINAL-Bench/VKORE-325M-NPU-Base")
gpu = AutoModelForCausalLM.from_pretrained("FINAL-Bench/VKORE-325M-NPU-Base", subfolder="b200_reference")

Model specification

Training data

• Dataset: HuggingFaceFW/fineweb-edu, sample/10BT (public)
• Tokenization: byte level, vocabulary 256
• Scale: 5.0 billion tokens
• The training data is fully public.

Training setup

• Hardware: commercial inference only NPU, 32 chips
• Reference: a B200 GPU run of the same model with the same initialization, data and schedule
• The core technique that enables stable backpropagation based training on an inference only NPU was developed in house and is under patent application.
• The training method will be disclosed in a forthcoming paper.

Generation example

Prompt: Photosynthesis is the process

Output (sampling, temperature 0.7):

Photosynthesis is the process by which plants convert sunlight into energy.

Usage

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("FINAL-Bench/VKORE-325M-NPU-Base")
prompt = "Photosynthesis is the process"
ids = torch.tensor([list(prompt.encode("utf-8"))])   # byte level: token id = byte value
out = model.generate(ids, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.15)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

A byte level tokenizer is used (byte values 0 to 255), so no tokenizer file is needed.

Scope

This is a base model (no instruction tuning) released as a research demonstration of from scratch pretraining on an inference only NPU.

Credits

VIDRAFT. Part of the VKORE project (NPU acceleration family). Published under FINAL-Bench.