squad-0.8b-MLX

A team of models. On your device. In milliseconds. ▶ Live demo (previous version)

squad-0.8b for Apple silicon, built from its October 2026 revision 58c22ba: the new trunk, a 1.1 MB router and four 2-bit LoRA experts (games, e2e, multilingual, documents), a MoQLE (Mixture of Quantized LoRA Experts). Same System One API, routing and answers as the CUDA version. Method and limitations: squad-0.8b.

Use

pip install mlx==0.32.4 mlx-lm==0.32.2 tokenizers decider-ai==1.9.0 huggingface_hub safetensors pillow numpy
hf download labqoat/squad-0.8b-MLX --local-dir squad-0.8b-MLX && cd squad-0.8b-MLX

from qev_mlx import QEVMLX   # qev_mlx.py in this repo

m = QEVMLX.load("labqoat/squad-0.8b-MLX")                 # bf16 trunk + router + 4 experts
m = QEVMLX.load("labqoat/squad-0.8b-MLX", subfolder="q8") # 8-bit trunk, the same experts
m = QEVMLX.load("labqoat/squad-0.8b-MLX", slots=False)    # the trunk alone
m.decide(state, questions)                                # images: "<image:NAME>" in the state + images={...}
m.last_slots                                              # the expert per prompt row (0 = trunk, 1..4)

python qev_mlx_server.py                     # POST http://127.0.0.1:8017/v1/systemone (prefix cache, batching, images)
python qev_mlx_server.py --subfolder q8      # the 8-bit trunk
python qev_mlx_server.py --no-slots          # the trunk alone

The response carries "routing": {"slots": [...], "names": [...]}, e.g. ["games"], ["multilingual"] or ["none"].

Routing is the CUDA runtime's: top-1 of {trunk, games, e2e, multilingual, documents} on the router's logits, and an expert listed in router.min_p_slots (manifest.json; e2e: 0.35) is taken only when the router gives it at least that probability, else the trunk answers. With encoding="per_question" every question of a request takes the route of the request's joint prompt. The router's 6 layers are the first 6 layers of the request's own forward (joint encoding), so text requests pay nothing extra. The experts stay 2-bit in memory (slots_2bit.safetensors, mx.quantized_matmul).

Results

Measured on the CUDA release (same weights; from the squad-0.8b card):

Checked against the CUDA reference (qev.QEV.from_pretrained(..., specialists=True) at 58c22ba, L40S) on 310 requests (387 questions: Benchmark Heaven public 231, game observations 30, browser steps 25, hand-written multilingual and document requests 24), on an M4 Pro:

Experts chosen by the CUDA reference on these requests: trunk 260, games 27, multilingual 17, e2e 5, documents 1. Every route that differs is a browser step whose e2e probability sits on the 0.35 threshold (0.347–0.352 here), where bf16 round-off decides; the larger |Δp| outliers are requests on a different side of that threshold.

Speed on an M4 Pro (24 GB), one request at a time, after warm-up, the same 310 requests:

Short requests (~30–70 tokens of state) take 34–53 ms; long browser states (~380 tokens) about 330 ms. For reference, the CUDA release on one L40S: 4.4 ms median on the same requests. Memory: about 2–2.5 GB for the bf16 model (mx.set_cache_limit keeps the MLX buffer cache small on a shared machine).

Image input was not re-evaluated for this revision: the trunk changed, the vision tower (vision.safetensors) is the same file as before.

Files

model.safetensors, vision.safetensors, configs and tokenizer (bf16; q8/ holds the 8-bit trunk, affine, group size 64) · manifest.json, slots_2bit.safetensors (10.9 MB, 4 experts), router.safetensors (1.1 MB): the experts, used by the root and by q8/ · qev_mlx.py, qev_mlx_server.py, qev_mlx_slots.py (converts the CUDA repo's specialists/ folder: python qev_mlx_slots.py SPECIALISTS OUT), qev_mlx_vision.py · SHA256SUMS.

Built from labqoat/squad-0.8b@58c22ba: the trunk is its model.safetensors in mlx-lm's Qwen3.5 layout (mlx-lm's own sanitize; the unchanged embeddings and final norm are bit-identical to the previous MLX build); the experts are its specialists/ rows re-packed for MLX, codes and per-row scales unchanged (max |dequantised difference| 1.5e-8).

License: Apache-2.0.