basal-1.5-4.5B-MLX-8bit

8-bit MLX version of basal-1.5-4.5B (4.5B) for Apple Silicon. Same model, same prompt format and the same typed answers (choice, noul, score) with a probability for every option; see the main card for the description, benchmarks, limitations and license.

Agreement with the bf16 model: 0.992; accuracy change +0.004 (0.940 vs 0.936) (500 held-out Polish development decisions, both option orders, calibrated, Apple M4 Pro; agreement = same top option as the reference (the same weights in PyTorch bf16; for basal-1.5-max, GGUF Q8_0 with llama.cpp); latency = median per decision (one question, both option orders) through the basal server path). Latency: 587 ms per decision on Apple M4 Pro, 24 GB. Memory: 5.2 GB. Accuracy here is on a 500-decision sample, to check that the port matches its reference; to compare model sizes, see the benchmark table of the main card.

Weights: MLX affine 8-bit decoder layers (group 64, a bf16 scale and bias per group); the token embeddings and the output head stay in bf16, because the decision is read from the output head's logits of the option letters. The recommended Apple Silicon port; -MLX-fp4 needs about half the memory.

Files: the MLX weights and config, tokenizer and chat template, CALIBRATION.json, basal.json.

Quick start

uv venv --python 3.12 ~/basal-mlx && source ~/basal-mlx/bin/activate
uv pip install "basal[mlx] @ https://github.com/rkinas/basal/archive/refs/tags/v1.5.0.tar.gz"
basal-serve --model Remek/basal-1.5-4.5B-MLX-8bit --mode mlx --port 8000

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": "Klient: od wczoraj nie mogę zalogować się do bankowości internetowej, system pokazuje błąd hasła.",
  "questions": {"dept": {"type": "choice", "instructions": "Do którego działu skierować zgłoszenie?",
    "criteria": {"cards": "Reklamacje kart", "online": "Wsparcie bankowości elektronicznej", "loans": "Kredyty"}}}}'

The basal server renders the model's own prompt, asks both option orders, applies the calibration and supports multi, act and "facts": "auto"; the state of a request is computed once into a KV cache shared by all its questions. Evidence spans are not available with MLX (use the bf16 model with the CUDA engine).

Engines

basal-serve is the decision layer in every row. vLLM, SGLang, MLX, Ollama and llama.cpp run the weights; the typed answers (a calibrated probability for each of your options, both option orders, the state shared by several questions, multi, act and facts) come from basal-serve --mode <engine> in front of them, with the same HTTP API in every mode. A plain ollama run or llama-cli gives free text only. Evidence spans need the PyTorch engine (fast, eager); vLLM and SGLang start with a 4,096-token context (longer states are refused); raise it with --max-len (e.g. 32768 for long documents). Install and first request for each engine: quick start.

Calibration

CALIBRATION.json carries the temperatures and confidence thresholds of the bf16 model. Checked on this port: the temperature that fits it best is 1.00 times the bf16 one per question type, so no separate calibration file is needed. The confidence thresholds are not certified for this port: refit them on your own labelled requests before automating decisions with them.

License

Apache-2.0, like basal-1.5-4.5B.