[!WARNING] These files need our llama.cpp fork. Upstream llama.cpp, Ollama, LM Studio and the other apps under "Use this model" do not support the
olalaarchitecture yet and cannot load them. Build the fork as shown in Run with llama.cpp.
GGUF files of Olala-v1.0-mini-7A1B for local inference with llama.cpp on CPUs, NVIDIA GPUs and Apple Silicon.
| File | Format | Size | Use it for |
|---|---|---|---|
Olala-v1.0-mini-7A1B-bf16.gguf |
BF16 (16-bit) | 13.6 GB | NVIDIA GPU; Mac with 24 GB or more |
Olala-v1.0-mini-7A1B-q8_0-safe2.gguf |
8-bit (q8_0-safe2) | 6.9 GB | x86 CPU; Mac with 16 GB |
Olala-v1.0-mini-7A1B-q6_k.gguf |
6-bit (Q6_K) | 6.3 GB | x86 CPU; Mac with 16 GB (fastest quantized file on Metal) |
Olala-v1.0-mini-7A1B-q5_k_m.gguf |
5-bit (Q5_K_M) | 6.0 GB | x86 CPU, Mac; smallest file with no measured quality loss |
Which one? BF16 if it fits. Otherwise q5_k_m (or q8_0-safe2 / q6_k if you have the memory); on a 16 GB Mac, q6_k.
On a Mac, also consider the MLX version: it decodes faster than llama.cpp on Apple Silicon.
There is no 4-bit file. A Q4_K_M file was built and tested, then withdrawn: it saves only 0.3 GB over q5_k_m and is measurably worse (MBPP+ 67.7, MGSM 65.1). It also reasons about 40% longer and runs into the context limit more than twice as often.
Quality of the quantized filesMeasured on Linux (x86 CPU and an NVIDIA H100). Benchmarks at temperature 0.5 and reasoning_effort medium; the
standard error is 2 to 3 points on every score.
| File | HumanEval+ | MBPP+ | MGSM |
|---|---|---|---|
| BF16 | 87.8 | 71.1 | 68.7 |
| q8_0-safe2 | 84.1 / 86.6 | 73.5 | 69.6 |
| q6_k | 87.8 / 86.0 | 74.8 | 68.7 |
| q5_k_m | 86.0 / 89.0 | 73.0 | 69.1 |
Measured on a MacBook Air M5 with 16 GB, macOS 26.5. All three quantized files load, answer a long prompt and make a tool call. BF16 does not fit in 16 GB.
| File | Prompt tokens/s | Generation tokens/s |
|---|---|---|
| q8_0-safe2 | 989 | 60 |
| q6_k | 932 | 72 |
| q5_k_m | 798 | 44 |
llama-bench, 512-token prompt and 128 generated tokens, with
other apps open, so they may be a little low.Upstream llama.cpp, Ollama and LM Studio do not support the olala architecture yet. Use the
llama.cpp fork, a fork of
ggml-org/llama.cpp.
Sampling parameters: temperature=0.8, top_p=0.9, top_k=40.
Reasoning effort: disabled, low, medium (default) or high. It is a chat-template argument
(reasoning_effort=... in apply_chat_template()).
[!TIP] Apple M1: run
export OLALA_NO_FUSED_MOE=1beforellama-cliorllama-server: generation is about 60 % faster (34 to 54 tokens/s on an M1 Pro). M2 and later need nothing.
1. Build the fork
git clone https://github.com/OVHai-LLM/llama.cpp.git llama.cpp-olala
cd llama.cpp-olala
# NVIDIA GPU (CUDA >= 12)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_USE_PREBUILT_UI=OFF
# Mac (Apple Silicon)
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_USE_PREBUILT_UI=OFF
# CPU only
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build -j
-DLLAMA_USE_PREBUILT_UI=OFF skips the download of the server's web UI, which does not match this fork yet.
The API is not affected.
2. Download a file
Pick the file for your hardware in the Files table. For example, on a Mac with 16 GB:
pip install -U huggingface_hub
hf download OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF Olala-v1.0-mini-7A1B-q6_k.gguf --local-dir .
The commands below name the file each setup uses; replace it with the one you downloaded.
3. Chat in the terminal
# x86 CPU
./build/bin/llama-cli -m Olala-v1.0-mini-7A1B-q8_0-safe2.gguf -t 16 -fa 1 -nr --temp 0.8 --top-p 0.9 --top-k 40 --reasoning-effort low
# NVIDIA GPU / Mac with >= 24 GB
./build/bin/llama-cli -m Olala-v1.0-mini-7A1B-bf16.gguf -ngl 99 -fa 1 --temp 0.8 --top-p 0.9 --top-k 40 --reasoning-effort low
# Mac with 16 GB
./build/bin/llama-cli -m Olala-v1.0-mini-7A1B-q6_k.gguf -ngl 99 -fa 1 -c 8192 --temp 0.8 --top-p 0.9 --top-k 40 --reasoning-effort low
Add -st -p "your question" for a single question instead of an interactive chat.
4. Serve an OpenAI-compatible API
# GPU
./build/bin/llama-server -m Olala-v1.0-mini-7A1B-bf16.gguf -ngl 99 -fa 1 -c 32768 -np 8 --host 0.0.0.0 --port 8080
# CPU
./build/bin/llama-server -m Olala-v1.0-mini-7A1B-q8_0-safe2.gguf -t 16 -fa 1 -nr -ctk q8_0 -ctv q8_0 -c 32768 -np 4 --port 8080
# Mac with 16 GB
./build/bin/llama-server -m Olala-v1.0-mini-7A1B-q6_k.gguf -ngl 99 -fa 1 -c 16384 --port 8080
curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "Explain how a heat pump works."}],
"temperature": 0.8, "top_p": 0.9, "top_k": 40,
"reasoning_effort": "medium"
}'
The response contains reasoning_content (the model's thinking), content (the answer) and tool_calls when
tools are passed.
-t: CPU threads; use at most the number of physical cores.-nr (no repack): x86 CPUs only; the x86 repacked kernels are slower on this model. On Arm CPUs (Apple
Silicon, Graviton) leave repacking on: there it is 1.4 to 4 times faster.-fa 1: flash attention, always on.-c: context size (at most 32,768), shared by the -np parallel slots. Prompts longer than -c fail
(recurrent models cannot shift context), so size it for your longest conversation times -np.-ngl 0) is slow with q8_0-safe2; prefer Metal (-ngl 99).OLALA_NO_FUSED_MOE=1 before llama-cli or llama-server. On an M1 Pro with 16 GB and
q5_k_m, generation goes from 34 to 54 tokens/s.Full setup guide: docs/olala-setup.md.
How the files were madeWith the llama.cpp fork, from the BF16 safetensors of Olala-v1.0-mini-7A1B:
python3 convert_hf_to_gguf.py Olala-v1.0-mini-7A1B --outtype bf16 --outfile Olala-v1.0-mini-7A1B-bf16.gguf
./build/bin/llama-quantize \
--tensor-type attn_output=bf16 --tensor-type ffn_down_exps=q6_k --tensor-type ffn_down_shexp=bf16 \
--tensor-type ffn_latent_up=bf16 --tensor-type ffn_up_exps=q8_0 \
Olala-v1.0-mini-7A1B-bf16.gguf Olala-v1.0-mini-7A1B-q8_0-safe2.gguf q8_0
The K-quant files use an importance matrix (wikitext-2 training split) and two overrides that every Olala 1.0 mini quantization needs: the 384-column tensors cannot be K-quantized and would silently fall back to legacy types.
./build/bin/llama-imatrix -m Olala-v1.0-mini-7A1B-bf16.gguf -f wiki.train.raw -o olala.imatrix --chunks 200
SAFE="--tensor-type ffn_latent_up=bf16 --tensor-type ffn_up_exps=q8_0"
./build/bin/llama-quantize --imatrix olala.imatrix $SAFE Olala-v1.0-mini-7A1B-bf16.gguf Olala-v1.0-mini-7A1B-q6_k.gguf q6_k
./build/bin/llama-quantize --imatrix olala.imatrix $SAFE Olala-v1.0-mini-7A1B-bf16.gguf Olala-v1.0-mini-7A1B-q5_k_m.gguf q5_k_m
A plain q8_0 (or other) quantization without these overrides can produce NaN on this model.
For the model itself (training, benchmarks, chat template, tool calling), see Olala-v1.0-mini-7A1B.
The uses and limitations of Olala-v1.0-mini-7A1B apply to these files. In addition:
Before you deploy: read the public summary of training content and our copyright policy.
| Licence | Apache License, Version 2.0 (SPDX: Apache-2.0) |
| Licensor | OVH LLM SAS, trading as OVHai LLM, 2 rue Kellermann, 59100 Roubaix, France |
| Copyright | Copyright 2026 OVH LLM SAS |
| Release date | 19 November 2026 |
| Distribution | Open weights |
| Notices | NOTICE: keep it when you redistribute the model or a derivative. It includes the attribution of the Qwen tokenizer (Copyright 2024 Alibaba Cloud, Apache 2.0) |
| Training content (EU AI Act) | Public summary of training content, on the European Commission template (Art. 53(1)(d)) |
@techreport{ovhaillm2026olala,
title = {Olala v1.0 mini Technical Report},
author = {{OVHai LLM Research Team}},
institution = {OVHcloud},
year = {2026},
url = {https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B/blob/main/Olala-v1.0-mini.pdf},
}
OVHaiLLM/Olala-v1.0-mini-7A1B