Nova v3

Nova v3 is a decoder-only causal language model with 148.8 million parameters. This repository provides the Transformers checkpoint, tokenizer, and custom model code required to load the model.

Model specifications

The model supports key/value caching during generation. Its custom implementation supports Transformers 4.57 and 5.x. Review the custom modeling files before loading with trust_remote_code=True.

Load with Transformers

Install PyTorch and Transformers:

pip install torch transformers

Load the tokenizer and model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "plasmova/nova-v3"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

system_prompt = (
    "You are Nova, a helpful and accurate assistant. Answer the user's request directly and concisely. "
    "For simple arithmetic, calculate the result and state it plainly. Do not invent names, scenarios, or equations."
)
prompt = (
    f"<|system|>{system_prompt}<|/system|>"
    "<|user|>Explain why the sky appears blue in one sentence.<|/user|>"
    "<|assistant|>"
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
output = model.generate(
    **inputs,
    max_new_tokens=160,
    do_sample=True,
    temperature=0.2,
    top_p=0.85,
    top_k=12,
    repetition_penalty=1.12,
    no_repeat_ngram_size=3,
    eos_token_id=[
        tokenizer.eos_token_id,
        tokenizer.convert_tokens_to_ids("<|/assistant|>"),
    ],
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Recommended starting settings are temperature 0.2, top-p 0.9, top-k 40, and repetition penalty 1.08. These focused settings are a starting point; adjust them for your task. The local inference script applies this concise system instruction by default. Recommended generation defaults are temperature 0.2, top-p 0.85, top-k 12, repetition penalty 1.12, no-repeat n-gram size 3, and up to 160 new tokens. Keep the total prompt and generated text within the 2,048-token context. The example runs on CPU; for GPU inference, move the model and inputs to a supported device and dtype.

Evaluation

The following zero-shot results use the first 500 examples from each task with lm-evaluation-harness 0.4.13 on CPU. These are limited samples, not full validation-set results.

Comparison with SmolLM2-135M

In a matched zero-shot run using 500 examples per task, Nova v3 scored 43.6% normalized accuracy on HellaSwag and 52.2% on WinoGrande, compared with 44.2% and 50.6% for SmolLM2-135M. The WinoGrande difference is small relative to sampling uncertainty; treat both results as early directional measurements, not established performance claims. The comparison details and per-run metrics are in benchmark_comparison.json.

On a separate 100-question GSM8K sample with five examples in context, Nova v3 and SmolLM2-135M-Instruct each answered 4 questions correctly (4.0% exact match). This small sample does not establish equivalence. Details are in gsm8k_comparison.json.

Training and license

The v3 training code specifies pretraining on a mixture of FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath, Wikipedia, and CodeParrot, followed by SmolTalk fine-tuning. Dataset contents remain subject to their respective upstream terms.

The model weights and accompanying custom model code are released under the Apache License 2.0. See LICENSE.