Olala-v1.0-mini-7A1B-base is the pre-trained base model of the Olala-v1.0-mini family, built on our hybrid Mamba-3 / attention mixture-of-experts architecture. It is not instruction-tuned.
[!WARNING] Pre-alignment model, toxic: do not use it as an assistant or expose it to end users. This base model had no instruction tuning and no alignment or safety training. It continues any text, including toxic, hateful, harmful, biased or false content, and does not refuse anything. Use it only for research and as a starting point for fine-tuning, with your own safety training and filters. For an assistant, use Olala-v1.0-mini-7A1B.
Find more information in our technical report.
The Olala-v1.0-mini family includes this base model, the SFT checkpoint, the post-trained model and two specialized models.
| Model | Description |
|---|---|
| Olala-v1.0-mini-7A1B-base | Pre-trained base model (12.8T tokens). Not aligned: research and fine-tuning only |
| Olala-v1.0-mini-7A1B-SFT | After supervised fine-tuning, before alignment (RL); can be toxic: research and fine-tuning only |
| Olala-v1.0-mini-7A1B | Final post-trained model: reasoning, chat and tool use |
| Olala-v1.0-summarise | Specialized for document summarization |
| Olala-v1.0-translate | Specialized for machine translation, including financial documents |
| Property | Specification |
|---|---|
| Checkpoint | Pre-trained base model, not instruction-tuned |
| Parameters | 6.8B total, about 1B active per token |
| Architecture | 36 layers: 29 Mamba-3 and 7 attention layers (one in five), each followed by a mixture-of-experts block with 256 routed experts (6 active) and 1 shared expert |
| Context length | 32K tokens (input + output)A 128K-context version is coming soon. |
| Vocabulary | 151,936 tokens |
| Training data | 12.8T tokens: public datasets and our own synthetic data |
| Languages | Pre-trained on English and 36 other languages (38 in annealing), mostly European, plus Arabic, Chinese, Japanese and Korean (full list in the technical report). Multilingual post-training evaluation covers 11 languages (MGSM) and 8 languages (Multi-IF). |
| License | Apache 2.0 |
Olala-v1.0-mini-7A1B-base is a starting point for fine-tuning, continued pre-training and research. For a chat assistant, use Olala-v1.0-mini-7A1B. For a ready-to-run fine-tuning example, see the fine-tuning image on Docker Hub and the OVHcloud guides to fine-tune with AI Training and deploy the result with AI Deploy.
Transformers quick start:
Requirements: an NVIDIA GPU, or the CPU. The model code runs in plain PyTorch, so a standard install is enough:
pip install torch "transformers==5.8.*" accelerate einops
Without extra packages, the Mamba-3 layers and the experts run in plain PyTorch: correct, but slow (on the CPU
alone, about 6 tokens/s on a 64-core server). For speed, also install the kernels the model code picks up by
itself: the Mamba-3 kernels (mamba_ssm) from the
mamba fork (branch dev), scattermoe
and flash_attn. They compile at install time (flash-attn alone can take an hour);
OVHai-LLM/olala-v1 pins a set that works.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "OVHaiLLM/Olala-v1.0-mini-7A1B-base"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16",
device_map="cuda") # or "cpu" without a GPU (slow)
inputs = tok("The three laws of thermodynamics are", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=200, use_cache=True)
print(tok.decode(out[0], skip_special_tokens=True))
use_cache=True: config.json sets use_cache: false.[!IMPORTANT] vLLM: always serve this model with
--mamba-ssm-cache-dtype float32. vLLM's default keeps the Mamba recurrent state in bf16, and long generations then degrade into repetition loops.
vLLM: serve it with the vLLM fork and
--mamba-ssm-cache-dtype float32, as for Olala-v1.0-mini-7A1B, and use the
completions endpoint (no chat template).
Olala-v1.0-mini-7A1B-base averages 57.1 over 13 benchmarks, the best score among mixture-of-experts models with about 1B active parameters. Mathematics is its strongest area and knowledge its weakest.
The table compares Olala-v1.0-mini-7A1B-base with Luciole-23B, the European base model of our comparison, with the two other mixture-of-experts models with about 1B active parameters, and with two recent Qwen3.5 models (4B and 2B). All 12 models are in the fold-out below.
| Benchmark | Olala v1.0-mini-7A1B-base 7B-A1B 12.8T tokens |
Luciole-23B 23B 5.0T tokens |
LFM2.5 8B-A1B 8B-A1B 38.0T tokens |
OLMoE-1B-7B 7B-A1B 5.1T tokens |
Qwen3.5-4B 4B 36.0T tokens |
Qwen3.5-2B 2B 36.0T tokens |
|---|---|---|---|---|---|---|
| Knowledge | ||||||
| MMLU | 57.5 | 64.3 | 63.2 | 55.1 | 73.3 | 54.5 |
| ARC-Challenge | 51.8 | 64.5 | 64.8 | 58.4 | 66.4 | 54.2 |
| ARC-Easy | 80.2 | 87.1 | 86.7 | 83.0 | 88.1 | 81.3 |
| GPQA | 31.0 | 31.9 | 32.8 | 23.4 | 39.1 | 29.9 |
| TriviaQA | 50.0 | 71.6 | 58.0 | 63.7 | 51.1 | 33.1 |
| Math | ||||||
| GSM8K | 66.0 | 56.1 | 67.2 | 45.4 | 83.0 | 60.6 |
| MATH | 72.2 | 31.8 | 46.0 | 17.1 | 54.4 | 35.8 |
| Multilingual | ||||||
| MGSM | 70.1 | 40.2 | 57.9 | n/a | 72.5 | 54.7 |
| Belebele* | 71.2 | 78.5 | 66.8 | 44.9 | 89.5 | 80.7 |
| Code | ||||||
| HumanEval | 42.1 | 37.2 | 55.5 | 11.0 | 50.6 | 32.9 |
| HumanEval+ | 38.4 | 32.9 | 50.0 | 11.0 | 44.5 | 29.3 |
| MBPP | 47.8 | 45.8 | 40.2 | 21.4 | 52.2 | 33.8 |
| MBPP+ | 67.2 | 65.6 | 56.1 | 28.8 | 71.7 | 48.9 |
| Long context | ||||||
| RULER 16K | 67.6 | 89.5 | 31.5 | n/a | 94.2 | 85.7 |
| RULER 32K* | 46.7 | 70.2 | 23.1 | n/a | 93.4 | 80.0 |
| Average 13 benchmarks, * excluded | 57.1 | 55.3 | 54.6 | 38.0 | 64.7 | 48.8 |
Every model was run by us with the LM Evaluation Harness (v0.4.11) against a vLLM server, through the same code path: same prompts, few-shot examples, answer extraction and scoring. Decoding is greedy.
MMLU 5-shot, ARC 25-shot (length-normalised), GPQA 0-shot
(main split): scored by log-likelihood.
TriviaQA 5-shot, GSM8K 5-shot, MATH 4-shot (Minerva prompts,
checked with math_verify), MGSM 8-shot over seven languages (English, German, French, Spanish,
Chinese, Japanese, Russian).
Belebele: 5-shot reading comprehension in 12 languages, added after the other benchmarks and
left out of the average.
HumanEval(+) 0-shot and MBPP(+) 3-shot: pass@1; the + variants use the EvalPlus
tests.
RULER: retrieval subtasks. RULER 32K is left out of the average so long context counts
once.
Average: unweighted mean of the 13 other benchmarks, over the benchmarks each model can run
(OLMoE-1B-7B has a 4K window and no MGSM score).
| Model | Params | Tokens | MMLU | ARC-C | ARC-E | GPQA | TrQA | GSM8K | MATH | MGSM | Bele* | HE | HE+ | MBPP | MBPP+ | R-16K | R-32K* | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Nemotron-3-Nano-30B | 30B-A3B | 25.0T | 78.5 | 67.4 | 89.5 | 40.0 | 76.9 | 85.1 | 83.0 | 81.5 | 88.7 | 74.4 | 70.1 | 70.0 | 84.9 | 97.0 | 96.5 | 76.8 |
| Qwen3.5-35B-A3B | 35B-A3B | 36.0T | 82.9 | 73.4 | 91.3 | 44.2 | 70.7 | 89.5 | 60.5 | 84.8 | 93.1 | 64.6 | 54.9 | 68.2 | 83.6 | 94.6 | 94.8 | 74.1 |
| Qwen3.5-9B | 9B | 36.0T | 72.4 | 70.6 | 89.6 | 37.5 | 58.7 | 87.0 | 55.6 | 80.1 | 91.3 | 61.6 | 50.0 | 61.4 | 76.2 | 94.6 | 94.1 | 68.9 |
| Qwen3.5-4B | 4B | 36.0T | 73.3 | 66.4 | 88.1 | 39.1 | 51.1 | 83.0 | 54.4 | 72.5 | 89.5 | 50.6 | 44.5 | 52.2 | 71.7 | 94.2 | 93.4 | 64.7 |
| Falcon-H1-3B | 3B | 2.5T | 68.3 | 61.5 | 85.0 | 33.7 | 41.3 | 70.0 | 49.6 | 69.3 | 82.2 | 57.3 | 53.0 | 58.8 | 74.6 | 50.8 | 40.6 | 59.5 |
| Olala-v1.0-mini-7A1B-base | 7B-A1B | 12.8T | 57.5 | 51.8 | 80.2 | 31.0 | 50.0 | 66.0 | 72.2 | 70.1 | 71.2 | 42.1 | 38.4 | 47.8 | 67.2 | 67.6 | 46.7 | 57.1 |
| Luciole-23B | 23B | 5.0T | 64.3 | 64.5 | 87.1 | 31.9 | 71.6 | 56.1 | 31.8 | 40.2 | 78.5 | 37.2 | 32.9 | 45.8 | 65.6 | 89.5 | 70.2 | 55.3 |
| LFM2.5-8B-A1B | 8B-A1B | 38.0T | 63.2 | 64.8 | 86.7 | 32.8 | 58.0 | 67.2 | 46.0 | 57.9 | 66.8 | 55.5 | 50.0 | 40.2 | 56.1 | 31.5 | 23.1 | 54.6 |
| SmolLM3-3B | 3B | 11.2T | 61.4 | 59.6 | 84.8 | 28.3 | 58.8 | 68.2 | 42.3 | 55.7 | 74.0 | 35.4 | 28.7 | 50.0 | 65.1 | 65.5 | 57.1 | 54.1 |
| Qwen3.5-2B | 2B | 36.0T | 54.5 | 54.2 | 81.3 | 29.9 | 33.1 | 60.6 | 35.8 | 54.7 | 80.7 | 32.9 | 29.3 | 33.8 | 48.9 | 85.7 | 80.0 | 48.8 |
| Qwen3.5-0.8B | 0.8B | 36.0T | 48.6 | 43.7 | 75.0 | 30.4 | 19.7 | 39.6 | 24.6 | 36.6 | 67.9 | 20.7 | 19.5 | 25.6 | 37.0 | 73.5 | 69.4 | 38.0 |
| OLMoE-1B-7B | 7B-A1B | 5.1T | 55.1 | 58.4 | 83.0 | 23.4 | 63.7 | 45.4 | 17.1 | n/a | 44.9 | 11.0 | 11.0 | 21.4 | 28.8 | n/a | n/a | 38.0 |
Pre-training: two phases build Olala-v1.0-mini-7A1B-base on 12.8T tokens, from public datasets and our own synthetic data.
Olala 1.0 mini was trained on three European supercomputers: JUPITER (Germany), Jean Zay (France) and MeluXina (Luxembourg). The base model was pre-trained on 1,024 GH200 GPUs of JUPITER. Its training emitted about 40 t CO₂e (market-based): see Environmental impact.
These figures cover the whole training of Olala 1.0 mini. This checkpoint is one of its stages (pre-training and annealing); the split by stage is not published.
Training
| Energy | About 1.16 GWh for the whole training: pre-training and annealing on JUPITER (1,024 GH200 GPUs, Germany), supervised fine-tuning and reinforcement learning on Jean Zay (H100 GPUs, France) and MeluXina (Luxembourg) |
| Emissions | About 40 t CO₂e (range 30 to 50 t), market-based: electricity covered by renewable supply contracts |
| Largest share | JUPITER, about 70% of the emissions |
Method: training emissions are energy × the carbon intensity of each site's electricity supply contract (market-based).
This is a pre-trained base model, for fine-tuning, continued pre-training and research.
If you modify this model substantially, or place a system built on it on the market under your own name, you may become a provider under the EU AI Act, with the obligations that come with it, including those for high-risk AI systems when your use case is listed in Annex III.
Before you deploy: read the public summary of training content and our copyright policy.
| Licence | Apache License, Version 2.0 (SPDX: Apache-2.0) |
| Licensor | OVH LLM SAS, trading as OVHai LLM, 2 rue Kellermann, 59100 Roubaix, France |
| Copyright | Copyright 2026 OVH LLM SAS |
| Release date | 19 November 2026 |
| Distribution | Open weights and open data |
| Notices | NOTICE: keep it when you redistribute the model or a derivative. It includes the attribution of the Qwen tokenizer (Copyright 2024 Alibaba Cloud, Apache 2.0) |
| Training content (EU AI Act) | Public summary of training content, on the European Commission template (Art. 53(1)(d)) |
@techreport{ovhaillm2026olala,
title = {Olala v1.0 mini Technical Report},
author = {{OVHai LLM Research Team}},
institution = {OVHcloud},
year = {2026},
url = {https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B/blob/main/Olala-v1.0-mini.pdf},
}