Novi Mini Base Preview 2

A second preview of Novi Mini Base by Novi AI.

Novi Mini Base is a compact language model pretraining project featuring a BananaMind 2-style architecture, using a downscaled Cagliostro v3 design as its architectural basis.

This repository contains an intermediate checkpoint from the ongoing pretraining run.

Model Information

Training Progress

This preview corresponds to checkpoint 321000.

Training is ongoing. The final model may differ from this preview.

Architecture

Novi Mini Base follows a BananaMind 2-style decoder-only Transformer architecture.

Its architectural basis is a scaled-down Cagliostro v3 design, adapted for a compact configuration.

Architecture Components

Model Dimensions

Triton Acceleration

The training script includes optional custom Triton kernels for selected operations.

The accelerated operations include:

• RMSNorm: Fused normalization forward and backward operations.
• SwiGLU: Fused activation and multiplication forward and backward operations.
• RoPE: Rotary position embedding forward and backward operations.

Attention continues to use PyTorch SDPA, while linear projections use the model's existing PyTorch layers.

Triton is optional. If it is unavailable or disabled, the model falls back to its PyTorch implementations.

Set NOVI_DISABLE_TRITON=1 to explicitly disable the Triton kernels.

Training

Novi Mini Base is pretrained from scratch using BananaAll.

The training script configures a total target of 15 billion tokens, divided into two training phases.

Training Configuration

The configured 457,764 optimizer steps assume the single-GPU batch configuration used for this run. The script calculates the planned step count from the token quotas, sequence length, batch size, and gradient accumulation.

Learning-Rate Schedule

The training schedule has three stages:

1. Warmup: The learning rate increases over the first 2,000 optimizer steps.
2. Stable: The learning rate remains at its configured maximum through the stable training phase.
3. Cosine cooldown: The learning rate gradually decreases during the final 15% of training.

The stable phase and cooldown phase each have their own dataset mixture.

Training Data

The training configuration uses five Hugging Face dataset repositories, with multiple configurations from some repositories.

Dataset Sources

All configured sources use their train split.

Stable Phase Dataset Weights

Cosine Cooldown Dataset Weights

The training worker streams dataset rows and samples from the configured mixture. Each phase receives its own token allocation, with the dataset weights determining the relative token quotas.

The training script also records the resolved dataset token allocations in dataset_tokens.json when the run completes.

Tokenizer

Novi Mini Base uses a byte-level BPE tokenizer trained from the configured pretraining dataset mixture.

The tokenizer is trained separately before model initialization. The model vocabulary is then configured to match the tokenizer.

Checkpoints

The training script automatically uploads checkpoints to Hugging Face every 500 optimizer steps.

It also supports automatic recovery by locating the newest complete remote checkpoint and resuming training from its saved state.

A complete recovery checkpoint includes model weights, optimizer state, scheduler state, trainer state, and random-number-generator state.

Checkpoint repository:

https://huggingface.co/Novi-AI/Novi-Mini-Base-Checkpoints

Loading the Model

Novi Mini Base uses a custom Transformers architecture.

The model repository includes custom configuration and modeling files. Loading may require trust_remote_code=True.

Only enable remote model code after reviewing and trusting the repository's implementation.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Novi-AI/Novi-Mini-Base-Preview-2"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model.eval()

prompt = "The future of small language models is"

inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

This is a basic text-continuation example. Novi Mini Base is a pretrained base model, not an instruction-tuned chatbot.

Intended Use

This is an experimental pretrained model intended for:

• Language-model experimentation.
• Fine-tuning experiments.
• Small-model research.
• Architecture evaluation.
• Testing lightweight language models.
• Studying compact autoregressive pretraining.

This preview is not intended to be treated as a finished production model.

Limitations

Novi Mini Base has approximately 9.97 million parameters, making it considerably smaller than most general-purpose language models.

It may:

• Produce repetitive or incoherent text.
• Generate inaccurate or fabricated information.
• Struggle with complex reasoning.
• Have limited factual knowledge.
• Fail to maintain consistency over longer sequences.
• Perform poorly on instruction-following tasks.

The model is a base pretrained checkpoint, so it is not specifically optimized for conversational instruction following.

Evaluation

No independent benchmark results are included in this preview.

Training progress and loss measurements should not be interpreted as benchmark scores or proof of model quality.

Project

Developed by Novi AI as part of Project Kairo.

• Preview model: Novi-AI/Novi-Mini-Base-Preview-2
• Main model: Novi-AI/Novi-Mini-Base
• Checkpoints: Novi-AI/Novi-Mini-Base-Checkpoints

Disclaimer

This repository contains an intermediate training preview. Performance, capabilities, and output quality are not representative of the final model.

The model is experimental and may produce inaccurate, incomplete, or nonsensical outputs.

---

Novi AI 2026 — Project Kairo