basal-1.5-4.5B-MLX-fp4
4-bit (nvfp4) MLX version of basal-1.5-4.5B (4.5B) for Apple Silicon. Same
model, same prompt format and the same typed answers (choice, noul, score) with a probability for every option;
see the main card for the description, benchmarks, limitations and license.
Agreement with the bf16 model: 0.970; accuracy change −0.014 (0.922 vs 0.936) (500 held-out Polish development decisions, both option orders, calibrated, Apple M4 Pro; agreement = same top option as the reference (the same weights in PyTorch bf16; for basal-1.5-max, GGUF Q8_0 with llama.cpp); latency = median per decision (one question, both option orders) through the basal server path). Latency: 581 ms per decision on Apple M4 Pro, 24 GB. Memory: 2.9 GB. Accuracy here is on a 500-decision sample, to check that the port matches its reference; to compare model sizes, see the benchmark table of the main card.
Weights: MLX nvfp4 decoder layers (4-bit floating point E2M1, group 16, FP8 scales); the token embeddings and the output head stay in bf16, because the decision is read from the
output head's logits of the option letters. Use it when memory is tight; -MLX-8bit changes fewer decisions.
Files: the MLX weights and config, tokenizer and chat template, CALIBRATION.json, basal.json.
Quick start
uv venv --python 3.12 ~/basal-mlx && source ~/basal-mlx/bin/activate
uv pip install "basal[mlx] @ https://github.com/rkinas/basal/archive/refs/tags/v1.5.0.tar.gz"
basal-serve --model Remek/basal-1.5-4.5B-MLX-fp4 --mode mlx --port 8000
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
"state": "Klient: od wczoraj nie mogę zalogować się do bankowości internetowej, system pokazuje błąd hasła.",
"questions": {"dept": {"type": "choice", "instructions": "Do którego działu skierować zgłoszenie?",
"criteria": {"cards": "Reklamacje kart", "online": "Wsparcie bankowości elektronicznej", "loans": "Kredyty"}}}}'
The basal server renders the model's own prompt, asks both option orders, applies the calibration and supports
multi, act and "facts": "auto"; the state of a request is computed once into a KV cache shared by all its
questions. Evidence spans are not available with MLX (use the bf16 model with the CUDA engine).
Engines
| engine | basal-serve mode |
hardware | what it is for | quick start | status (basal-1.5, 4.5B) |
|---|---|---|---|---|---|
| basal engine (PyTorch) | fast (also fast-nocompile, fp8, nvfp4); eager --device mps|cpu |
NVIDIA GPUs (sm80+); Apple Silicon (MPS) and CPU with eager |
the primary server: packed requests (SOAM), evidence spans, fp8 on Hopper and Blackwell |
basal-serve --model Remek/basal-1.5-4.5B --mode fast |
verified: served check on one H100, 13.7 ms p50 per question over HTTP, 64.4 decisions/s with 32 clients |
| vLLM | vllm |
NVIDIA GPUs | an alternative CUDA server | basal-serve --model Remek/basal-1.5-4.5B --mode vllm (extra basal[vllm]) |
verified offline (one H100, basal-bench, 1,000 development decisions): agreement 0.994 with the fp32 engine, accuracy 0.928 vs 0.930, 147 decisions/s |
| SGLang | sglang |
NVIDIA GPUs | an alternative CUDA server (RadixAttention prefix sharing) | basal-serve --model Remek/basal-1.5-4.5B --mode sglang (own environment: sglang[srt]==0.5.21, then basal with --no-deps) |
verified offline (one H100, basal-bench, 1,000 development decisions): agreement 0.995 with the fp32 engine, accuracy 0.925 vs 0.930, 104.6 decisions/s; on Werdykt the same answer as the basal engine on 98.3% of items (97.0% for states over 8k tokens) |
| MLX | mlx |
Apple Silicon | Macs: -MLX-8bit (recommended) or -MLX-fp4 (half the memory) |
basal-serve --model Remek/basal-1.5-4.5B-MLX-8bit --mode mlx (extra basal[mlx]) |
verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.992 (-MLX-8bit) / 0.970 (-MLX-fp4), 587 / 581 ms per question |
| Ollama | ollama |
CPU, Apple Silicon, consumer GPUs | local and desktop use with Ollama (-GGUF) |
1. hf download Remek/basal-1.5-4.5B-GGUF --local-dir basal-1.5-4.5B-GGUF 2. in that folder: ollama create basal-1.5-4.5b:q8_0 -f Modelfile.Q8_0 3. basal-serve --model ./basal-1.5-4.5B-GGUF --mode ollama --ollama-model basal-1.5-4.5b:q8_0 |
verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.994 (Q8_0) / 0.980 (Q4_K_M), 518 / 542 ms per question |
| llama.cpp | llamacpp |
CPU, Apple Silicon, consumer GPUs | llama-server with the -GGUF files; the engine's token ids are sent as is |
1. llama-server -m basal-1.5-4.5B-GGUF/basal-1.5-4.5B-Q8_0.gguf -c 4096 --port 8080 2. basal-serve --model ./basal-1.5-4.5B-GGUF --mode llamacpp |
verified on an Apple M4 Pro (500 development decisions): agreement with bf16 0.994 (Q8_0) / 0.980 (Q4_K_M), 427 / 446 ms per question |
basal-serve is the decision layer in every row. vLLM, SGLang, MLX, Ollama and llama.cpp run the weights; the typed answers (a calibrated probability for each of your options, both option orders, the state shared by several questions, multi, act and facts) come from basal-serve --mode <engine> in front of them, with the same HTTP API in every mode. A plain ollama run or llama-cli gives free text only. Evidence spans need the PyTorch engine (fast, eager); vLLM and SGLang start with a 4,096-token context (longer states are refused); raise it with --max-len (e.g. 32768 for long documents). Install and first request for each engine: quick start.
Calibration
CALIBRATION.json carries the temperatures and confidence thresholds of the bf16 model. Checked on this port: per question type, the temperature that fits it best is 0.93–0.94 times the bf16 one, and refitting lowers the KL divergence by at most 4%, so no separate calibration file is shipped. The confidence thresholds are not certified for this port: refit them on your own labelled requests before automating decisions with them.
License
Apache-2.0, like basal-1.5-4.5B.