[!WARNING] These weights need our mlx-engine fork. Upstream mlx-lm and LM Studio do not support the
olalaarchitecture yet and cannot load them. Install the fork as shown in Run on a Mac.
MLX versions of Olala-v1.0-mini-7A1B for local inference on Apple Silicon Macs.
Each folder is a complete MLX model (weights, config, tokenizer, chat template). Download only the one you need.
| Folder | Quantization | Bits / weight | Size |
|---|---|---|---|
bf16/ |
None (BF16, as in the source checkpoint) | 16 | 13.6 GB |
q8/ |
8-bit, group 64 | 8.56 | 7.2 GB |
q5e6/ |
Routed experts 5-bit, everything else 6-bit, group 64 | 5.76 | 4.9 GB |
q4e8/ |
Routed experts 4-bit (group 32), everything else 8-bit (group 64) | 5.75 | 4.9 GB |
Upstream mlx-lm, mlx-vlm and LM Studio do not support the olala architecture yet. Use the
mlx-engine fork (a fork of LM Studio's
mlx-engine), with Python 3.11 on macOS 14 or later. The Mamba
state is kept in float32 automatically; there is no flag to set.
Sampling parameters: temperature=0.8, top_p=0.9, top_k=40.
Reasoning effort: disabled, low, medium (default) or high. It is a chat-template argument
(reasoning_effort=... in apply_chat_template()). In the server, pass it inside chat_template_kwargs.
[!WARNING] Apple M1: run
export OLALA_MOE_V3=0before Python or the server. Without it, M1 Macs crash on the first generated token ("Thread group size … > 384"). M2 and later need nothing.
1. Install the fork and download a folder
git clone https://github.com/OVHai-LLM/mlx-engine.git
cd mlx-engine
python3.11 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/hf download OVHaiLLM/Olala-v1.0-mini-7A1B-MLX --include "q5e6/*" --local-dir ./olala-mlx
2. Chat from Python (mlx-lm)
Run from the mlx-engine checkout:
from mlx_engine.models.olala import register
register() # registers the olala architecture with mlx-lm
from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler
model, tok = load("./olala-mlx/q5e6")
prompt = tok.apply_chat_template(
[{"role": "user", "content": "Explain how a heat pump works."}],
add_generation_prompt=True, reasoning_effort="medium",
)
for r in stream_generate(model, tok, prompt, max_tokens=4096,
sampler=make_sampler(temp=0.8, top_p=0.9, top_k=40)):
print(r.text, end="", flush=True)
This prints the raw output with its channels (reasoning in analysis, the answer in final, tool calls in
tools). For a single question from the terminal: .venv/bin/python demo.py --model ./olala-mlx/q5e6 --no-system --prompt "..." (--no-system because the chat template already has its own system prompt).
3. Serve an OpenAI-compatible API
export MLX_ENGINE_API_KEY=choose-a-key # required; clients send it as a Bearer token
.venv/bin/python -m mlx_engine.server --model ./olala-mlx/q5e6 \
--host 127.0.0.1 --port 8000 --context-length 8192 --parallel-sessions 1
All these options are required. --context-length goes up to 32,768 and is shared by the
--parallel-sessions; raise them if you have the memory. Use --host 0.0.0.0 to reach the server from other
machines. The server holds one model.
curl -sN http://127.0.0.1:8000/v1/chat/completions \
-H "Authorization: Bearer $MLX_ENGINE_API_KEY" -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "Explain how a heat pump works."}],
"stream": true, "temperature": 0.8, "top_p": 0.9, "top_k": 40, "max_tokens": 4096,
"chat_template_kwargs": {"reasoning_effort": "medium"}
}'
With the OpenAI Python client, top_k and chat_template_kwargs go through extra_body:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="choose-a-key")
tools = [{"type": "function", "function": {
"name": "get_weather", "description": "Get the current weather for a city.",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
stream = client.chat.completions.create(
model="olala", # ignored: the server holds one model
messages=[{"role": "user", "content": "What's the weather in Lyon?"}],
tools=tools, stream=True, temperature=0.8, top_p=0.9, max_tokens=4096,
extra_body={"top_k": 40, "chat_template_kwargs": {"reasoning_effort": "low"}},
)
for chunk in stream:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
if getattr(delta, "reasoning_content", None):
print(delta.reasoning_content, end="", flush=True)
if delta.content:
print(delta.content, end="", flush=True)
for call in delta.tool_calls or []:
print("\ntool call:", call.id, call.function.name, call.function.arguments)
# tool call: call_0 get_weather {"city": "Lyon"}
To continue after a tool call, send the conversation back with the assistant message (and its tool_calls) and
a {"role": "tool", "tool_call_id": ..., "name": ..., "content": ...} message.
"stream": true, temperature and top_k are required.chat_template_kwargs.reasoning_content (the reasoning), then content (the answer). Tool calls arrive as one
tool_calls delta at the end of the turn, with finish_reason: "tool_calls".tool_choice accepts "auto" and "none" only. seed, logprobs, logit_bias and the presence and
frequency penalties are rejected with a 400 error.max_tokens, the stream ends with an error event
(MLX generation ended without a stop condition.) instead of finish_reason: "length". The text sent before
it is valid.GET /health with the same Bearer token.With the fork, from the BF16 safetensors of Olala-v1.0-mini-7A1B:
python -m mlx_engine.models.olala.convert --hf-path Olala-v1.0-mini-7A1B --mlx-path q5e6 --recipe q5e6
Recipes: bf16, q8, q6, q5e6, q4e8, q4. Plain 4-bit (q4) loses too much quality and is not published. bf16/ holds the source weights unchanged; projections that read the same input are concatenated at
conversion, so the folders do not need the Hugging Face remote code.
The uses and limitations of Olala-v1.0-mini-7A1B apply to these files. In addition:
OLALA_MOE_V3=0 before running Python or the server. On an M1 Pro with 16 GB and q5e6, the card's
examples then run at about 62 tokens/s.Before you deploy: read the public summary of training content and our copyright policy.
| Licence | Apache License, Version 2.0 (SPDX: Apache-2.0) |
| Licensor | OVH LLM SAS, trading as OVHai LLM, 2 rue Kellermann, 59100 Roubaix, France |
| Copyright | Copyright 2026 OVH LLM SAS |
| Release date | 19 November 2026 |
| Distribution | Open weights |
| Notices | NOTICE: keep it when you redistribute the model or a derivative. It includes the attribution of the Qwen tokenizer (Copyright 2024 Alibaba Cloud, Apache 2.0) |
| Training content (EU AI Act) | Public summary of training content, on the European Commission template (Art. 53(1)(d)) |
@techreport{ovhaillm2026olala,
title = {Olala v1.0 mini Technical Report},
author = {{OVHai LLM Research Team}},
institution = {OVHcloud},
year = {2026},
url = {https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B/blob/main/Olala-v1.0-mini.pdf},
}
q8/model-00001-of-00002.safetensors 5.3 GBbf16/model-00002-of-00003.safetensors 5.3 GBbf16/model-00001-of-00003.safetensors 5.2 GBq5e6/model.safetensors 4.9 GBq4e8/model.safetensors 4.9 GBbf16/model-00003-of-00003.safetensors 3.1 GBq8/model-00002-of-00002.safetensors 1.9 GB