Olala-v1.0-mini-7A1B-base
Olala-v1.0-mini-7A1B-base is the pre-trained base model of the Olala-v1.0-mini family, built on our hybrid Mamba-3 / attention mixture-of-experts architecture. It is not instruction-tuned.
[!WARNING] Pre-alignment model, toxic: do not use it as an assistant or expose it to end users. This base model had no instruction tuning and no alignment or safety training. It continues any text, including toxic, hateful, harmful, biased or false content, and does not refuse anything. Use it only for research and as a starting point for fine-tuning, with your own safety training and filters. For an assistant, use Olala-v1.0-mini-7A1B.
- Open data: 12.8T tokens of public datasets and our own synthetic data, in English and 36 other languages (38 in annealing). The data recipe is listed per source in the technical report.
- Strong at mathematics: 72.2 on MATH (4-shot), the second-best score of the 12 base models we compared, ahead of every Qwen3.5 model including Qwen3.5-35B-A3B.
- Best in its class: among mixture-of-experts models with about 1B active parameters, it has the highest average, ahead of LFM2.5-8B-A1B, which was trained on three times more tokens. It is also ahead of the French model Luciole-23B, which has three times more parameters.
- Made in Europe: designed in France by OVHai LLM (OVHcloud group) and trained on three European supercomputers (JUPITER, Jean Zay and MeluXina), with open weights under Apache 2.0.
Find more information in our technical report.
Model details
The Olala-v1.0-mini family includes this base model, the SFT checkpoint, the post-trained model and two specialized models.
| Model | Description |
|---|---|
| Olala-v1.0-mini-7A1B-base | Pre-trained base model (12.8T tokens). Not aligned: research and fine-tuning only |
| Olala-v1.0-mini-7A1B-SFT | After supervised fine-tuning, before alignment (RL); can be toxic: research and fine-tuning only |
| Olala-v1.0-mini-7A1B | Final post-trained model: reasoning, chat and tool use |
| Olala-v1.0-summarise | Specialized for document summarization |
| Olala-v1.0-translate | Specialized for machine translation, including financial documents |
| Property | Specification |
|---|---|
| Checkpoint | Pre-trained base model, not instruction-tuned |
| Parameters | 6.8B total, about 1B active per token |
| Architecture | 36 layers: 29 Mamba-3 and 7 attention layers (one in five), each followed by a mixture-of-experts block with 256 routed experts (6 active) and 1 shared expert |
| Context length | 32K tokens (input + output)A 128K-context version is coming soon. |
| Vocabulary | 151,936 tokens |
| Training data | 12.8T tokens: public datasets and our own synthetic data |
| Languages | Pre-trained on English and 36 other languages (38 in annealing), mostly European, plus Arabic, Chinese, Japanese and Korean (full list in the technical report). Multilingual post-training evaluation covers 11 languages (MGSM) and 8 languages (Multi-IF). |
| License | Apache 2.0 |
Olala-v1.0-mini-7A1B-base is a starting point for fine-tuning, continued pre-training and research. For a chat assistant, use Olala-v1.0-mini-7A1B. For a ready-to-run fine-tuning example, see the fine-tuning image on Docker Hub and the OVHcloud guides to fine-tune with AI Training and deploy the result with AI Deploy.
Inference
Transformers quick start:
Requirements: an NVIDIA GPU, or the CPU. The model code runs in plain PyTorch, so a standard install is enough:
pip install torch "transformers==5.8.*" accelerate einops
Without extra packages, the Mamba-3 layers and the experts run in plain PyTorch: correct, but slow (on the CPU
alone, about 6 tokens/s on a 64-core server). For speed, also install the kernels the model code picks up by
itself: the Mamba-3 kernels (mamba_ssm) from the
mamba fork (branch dev), scattermoe
and flash_attn. They compile at install time (flash-attn alone can take an hour);
OVHai-LLM/olala-v1 pins a set that works.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "OVHaiLLM/Olala-v1.0-mini-7A1B-base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16",
device_map="cuda") # or "cpu" without a GPU (slow)
inputs = tok("The three laws of thermodynamics are", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=200, use_cache=True)
print(tok.decode(out[0], skip_special_tokens=True))
- Pass
use_cache=True:config.jsonsetsuse_cache: false. - Generate one prompt at a time. Padded batches are not handled in cached decoding.
[!IMPORTANT] vLLM: always serve this model with
--mamba-ssm-cache-dtype float32. vLLM's default keeps the Mamba recurrent state in bf16, and long generations then degrade into repetition loops.
vLLM: serve it with the vLLM fork and
--mamba-ssm-cache-dtype float32, as for Olala-v1.0-mini-7A1B, and use the
completions endpoint (no chat template).
Performance
Benchmarks
Olala-v1.0-mini-7A1B-base averages 57.1 over 13 benchmarks, the best score among mixture-of-experts models with about 1B active parameters. Mathematics is its strongest area and knowledge its weakest.
The table compares Olala-v1.0-mini-7A1B-base with Luciole-23B, the European base model of our comparison, with the two other mixture-of-experts models with about 1B active parameters, and with two recent Qwen3.5 models (4B and 2B). All 12 models are in the fold-out below.
| Benchmark | Olala v1.0-mini-7A1B-base 7B-A1B 12.8T tokens |
Luciole-23B 23B 5.0T tokens |
LFM2.5 8B-A1B 8B-A1B 38.0T tokens |
OLMoE-1B-7B 7B-A1B 5.1T tokens |
Qwen3.5-4B 4B 36.0T tokens |
Qwen3.5-2B 2B 36.0T tokens |
|---|---|---|---|---|---|---|
| Knowledge | ||||||
| MMLU | 57.5 | 64.3 | 63.2 | 55.1 | 73.3 | 54.5 |
| ARC-Challenge | 51.8 | 64.5 | 64.8 | 58.4 | 66.4 | 54.2 |
| ARC-Easy | 80.2 | 87.1 | 86.7 | 83.0 | 88.1 | 81.3 |
| GPQA | 31.0 | 31.9 | 32.8 | 23.4 | 39.1 | 29.9 |
| TriviaQA | 50.0 | 71.6 | 58.0 | 63.7 | 51.1 | 33.1 |
| Math | ||||||
| GSM8K | 66.0 | 56.1 | 67.2 | 45.4 | 83.0 | 60.6 |
| MATH | 72.2 | 31.8 | 46.0 | 17.1 | 54.4 | 35.8 |
| Multilingual | ||||||
| MGSM | 70.1 | 40.2 | 57.9 | n/a | 72.5 | 54.7 |
| Belebele* | 71.2 | 78.5 | 66.8 | 44.9 | 89.5 | 80.7 |
| Code | ||||||
| HumanEval | 42.1 | 37.2 | 55.5 | 11.0 | 50.6 | 32.9 |
| HumanEval+ | 38.4 | 32.9 | 50.0 | 11.0 | 44.5 | 29.3 |
| MBPP | 47.8 | 45.8 | 40.2 | 21.4 | 52.2 | 33.8 |
| MBPP+ | 67.2 | 65.6 | 56.1 | 28.8 | 71.7 | 48.9 |
| Long context | ||||||
| RULER 16K | 67.6 | 89.5 | 31.5 | n/a | 94.2 | 85.7 |
| RULER 32K* | 46.7 | 70.2 | 23.1 | n/a | 93.4 | 80.0 |
| Average 13 benchmarks, * excluded | 57.1 | 55.3 | 54.6 | 38.0 | 64.7 | 48.8 |
Every model was run by us with the LM Evaluation Harness (v0.4.11) against a vLLM server, through the same code path: same prompts, few-shot examples, answer extraction and scoring. Decoding is greedy.
MMLU 5-shot, ARC 25-shot (length-normalised), GPQA 0-shot
(main split): scored by log-likelihood.
TriviaQA 5-shot, GSM8K 5-shot, MATH 4-shot (Minerva prompts,
checked with math_verify), MGSM 8-shot over seven languages (English, German, French, Spanish,
Chinese, Japanese, Russian).
Belebele: 5-shot reading comprehension in 12 languages, added after the other benchmarks and
left out of the average.
HumanEval(+) 0-shot and MBPP(+) 3-shot: pass@1; the + variants use the EvalPlus
tests.
RULER: retrieval subtasks. RULER 32K is left out of the average so long context counts
once.
Average: unweighted mean of the 13 other benchmarks, over the benchmarks each model can run
(OLMoE-1B-7B has a 4K window and no MGSM score).
| Model | Params | Tokens | MMLU | ARC-C | ARC-E | GPQA | TrQA | GSM8K | MATH | MGSM | Bele* | HE | HE+ | MBPP | MBPP+ | R-16K | R-32K* | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Nemotron-3-Nano-30B | 30B-A3B | 25.0T | 78.5 | 67.4 | 89.5 | 40.0 | 76.9 | 85.1 | 83.0 | 81.5 | 88.7 | 74.4 | 70.1 | 70.0 | 84.9 | 97.0 | 96.5 | 76.8 |
| Qwen3.5-35B-A3B | 35B-A3B | 36.0T | 82.9 | 73.4 | 91.3 | 44.2 | 70.7 | 89.5 | 60.5 | 84.8 | 93.1 | 64.6 | 54.9 | 68.2 | 83.6 | 94.6 | 94.8 | 74.1 |
| Qwen3.5-9B | 9B | 36.0T | 72.4 | 70.6 | 89.6 | 37.5 | 58.7 | 87.0 | 55.6 | 80.1 | 91.3 | 61.6 | 50.0 | 61.4 | 76.2 | 94.6 | 94.1 | 68.9 |
| Qwen3.5-4B | 4B | 36.0T | 73.3 | 66.4 | 88.1 | 39.1 | 51.1 | 83.0 | 54.4 | 72.5 | 89.5 | 50.6 | 44.5 | 52.2 | 71.7 | 94.2 | 93.4 | 64.7 |
| Falcon-H1-3B | 3B | 2.5T | 68.3 | 61.5 | 85.0 | 33.7 | 41.3 | 70.0 | 49.6 | 69.3 | 82.2 | 57.3 | 53.0 | 58.8 | 74.6 | 50.8 | 40.6 | 59.5 |
| Olala-v1.0-mini-7A1B-base | 7B-A1B | 12.8T | 57.5 | 51.8 | 80.2 | 31.0 | 50.0 | 66.0 | 72.2 | 70.1 | 71.2 | 42.1 | 38.4 | 47.8 | 67.2 | 67.6 | 46.7 | 57.1 |
| Luciole-23B | 23B | 5.0T | 64.3 | 64.5 | 87.1 | 31.9 | 71.6 | 56.1 | 31.8 | 40.2 | 78.5 | 37.2 | 32.9 | 45.8 | 65.6 | 89.5 | 70.2 | 55.3 |
| LFM2.5-8B-A1B | 8B-A1B | 38.0T | 63.2 | 64.8 | 86.7 | 32.8 | 58.0 | 67.2 | 46.0 | 57.9 | 66.8 | 55.5 | 50.0 | 40.2 | 56.1 | 31.5 | 23.1 | 54.6 |
| SmolLM3-3B | 3B | 11.2T | 61.4 | 59.6 | 84.8 | 28.3 | 58.8 | 68.2 | 42.3 | 55.7 | 74.0 | 35.4 | 28.7 | 50.0 | 65.1 | 65.5 | 57.1 | 54.1 |
| Qwen3.5-2B | 2B | 36.0T | 54.5 | 54.2 | 81.3 | 29.9 | 33.1 | 60.6 | 35.8 | 54.7 | 80.7 | 32.9 | 29.3 | 33.8 | 48.9 | 85.7 | 80.0 | 48.8 |
| Qwen3.5-0.8B | 0.8B | 36.0T | 48.6 | 43.7 | 75.0 | 30.4 | 19.7 | 39.6 | 24.6 | 36.6 | 67.9 | 20.7 | 19.5 | 25.6 | 37.0 | 73.5 | 69.4 | 38.0 |
| OLMoE-1B-7B | 7B-A1B | 5.1T | 55.1 | 58.4 | 83.0 | 23.4 | 63.7 | 45.4 | 17.1 | n/a | 44.9 | 11.0 | 11.0 | 21.4 | 28.8 | n/a | n/a | 38.0 |
Training
Pre-training: two phases build Olala-v1.0-mini-7A1B-base on 12.8T tokens, from public datasets and our own synthetic data.
- Main pre-training (10.4T tokens): English web (Nemotron-CC v2), multilingual web in 36 languages (FineWeb-2), code, mathematics and translation pairs.
- Annealing (2.38T tokens): the mix moves to curated data. Mathematics becomes the largest share (36%), instruction data is added, and the multilingual data is enriched with Wikipedia and synthetic study notes. Three of these synthetic datasets are released: Math-Enrich-Multilingual, Wikipedia-ActiveReading and ProofWiki-ActiveReading, with the code that generated them.
Olala 1.0 mini was trained on three European supercomputers: JUPITER (Germany), Jean Zay (France) and MeluXina (Luxembourg). The base model was pre-trained on 1,024 GH200 GPUs of JUPITER. Its training emitted about 40 t CO₂e (market-based): see Environmental impact.
Environmental impact
These figures cover the whole training of Olala 1.0 mini. This checkpoint is one of its stages (pre-training and annealing); the split by stage is not published.
Training
| Energy | About 1.16 GWh for the whole training: pre-training and annealing on JUPITER (1,024 GH200 GPUs, Germany), supervised fine-tuning and reinforcement learning on Jean Zay (H100 GPUs, France) and MeluXina (Luxembourg) |
| Emissions | About 40 t CO₂e (range 30 to 50 t), market-based: electricity covered by renewable supply contracts |
| Largest share | JUPITER, about 70% of the emissions |
Method: training emissions are energy × the carbon intensity of each site's electricity supply contract (market-based).
Uses and limitations
This is a pre-trained base model, for fine-tuning, continued pre-training and research.
- Not an assistant. It was not trained to follow instructions, call tools or refuse harmful requests. Do not expose it to end users as is.
- Pre-alignment, toxic: no alignment or safety training; it can produce toxic, harmful, biased or false content.
- Knowledge is limited and can be wrong; the limitations of Olala-v1.0-mini-7A1B on languages, context and quantization also apply.
If you modify this model substantially, or place a system built on it on the market under your own name, you may become a provider under the EU AI Act, with the obligations that come with it, including those for high-risk AI systems when your use case is listed in Annex III.
Before you deploy: read the public summary of training content and our copyright policy.
License
| Licence | Apache License, Version 2.0 (SPDX: Apache-2.0) |
| Licensor | OVH LLM SAS, trading as OVHai LLM, 2 rue Kellermann, 59100 Roubaix, France |
| Copyright | Copyright 2026 OVH LLM SAS |
| Release date | 19 November 2026 |
| Distribution | Open weights and open data |
| Notices | NOTICE: keep it when you redistribute the model or a derivative. It includes the attribution of the Qwen tokenizer (Copyright 2024 Alibaba Cloud, Apache 2.0) |
| Training content (EU AI Act) | Public summary of training content, on the European Commission template (Art. 53(1)(d)) |
Citation
@techreport{ovhaillm2026olala,
title = {Olala v1.0 mini Technical Report},
author = {{OVHai LLM Research Team}},
institution = {OVHcloud},
year = {2026},
url = {https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B/blob/main/Olala-v1.0-mini.pdf},
}