basal-1.5-4.5B-GGUF

GGUF version of basal-1.5-4.5B (4.5B) for Ollama and llama.cpp: CPUs, Apple Silicon and consumer GPUs. See the main card for the description, benchmarks, limitations and license.

Accuracy here is on a 500-decision sample, to check that each file matches its reference; to compare model sizes, see the benchmark table of the main card.

Latency: 427 ms with llama.cpp and 518 ms with Ollama (Q8_0); 446 / 542 ms (Q4_K_M) per decision (Apple M4 Pro, 24 GB, Metal; llama.cpp 0.5.0, Ollama 0.35.0).

Converted with llama.cpp convert_hf_to_gguf.py and llama-quantize, with two fixes that matter for decisions:

• Tokenizer fields. The stock conversion writes the same score (−1000) for every token of this tokenizer, so Ollama and other text clients would split prompts differently from the original tokenizer. This export sets the scores from the BPE merge ranks, sets add_space_prefix = false and matches every added token (<|im_start|>, <|im_end|>, …) whole, as the Hugging Face tokenizer does, so prompts are tokenized exactly as by the original tokenizer and Ollama and llama.cpp give the same answers.
• Q4_K_M is quantised with an importance matrix computed on decision prompts and keeps the output head and the token embeddings at Q8_0 (the decision is read from the output head's logits of the option letters).

Quick start (Ollama)

basal is a typed-decision model, not a chat model: the basal server runs in front of Ollama, renders the model's own prompt and answer prefix, sends it raw and reads the log-probabilities of the option letters of one generated token. Ollama 0.12.11 or newer.

hf download Remek/basal-1.5-4.5B-GGUF --local-dir basal-1.5-4.5B-GGUF
cd basal-1.5-4.5B-GGUF && ollama create basal-1.5-4.5b:q8_0 -f Modelfile.Q8_0 && cd ..
uv pip install "basal @ https://github.com/rkinas/basal/archive/refs/tags/v1.5.0.tar.gz"
basal-serve --model ./basal-1.5-4.5B-GGUF --mode ollama --ollama-model basal-1.5-4.5b:q8_0 --port 8000

--model gives the tokenizer, chat template and calibration file (this folder or Remek/basal-1.5-4.5B-GGUF); Ollama serves the weights. Then send requests to http://127.0.0.1:8000/v1/systemone as with the bf16 model (curl example on the main card).

llama.cpp (llama-server), with the engine's token ids sent directly:

llama-server -m basal-1.5-4.5B-GGUF/basal-1.5-4.5B-Q8_0.gguf -c 4096 --port 8080
basal-serve --model ./basal-1.5-4.5B-GGUF --mode llamacpp --llamacpp-url http://127.0.0.1:8080 --port 8000

Engines

basal-serve is the decision layer in every row. vLLM, SGLang, MLX, Ollama and llama.cpp run the weights; the typed answers (a calibrated probability for each of your options, both option orders, the state shared by several questions, multi, act and facts) come from basal-serve --mode <engine> in front of them, with the same HTTP API in every mode. A plain ollama run or llama-cli gives free text only. Evidence spans need the PyTorch engine (fast, eager); vLLM and SGLang start with a 4,096-token context (longer states are refused); raise it with --max-len (e.g. 32768 for long documents). Install and first request for each engine: quick start.

What differs from the bf16 model in the basal engine

• Same prompt, both option orders, multi, act and "facts": "auto"; the state of a request is computed once by the runtime's prompt cache (requests of one state are sent one after another; --http-parallel N with OLLAMA_NUM_PARALLEL / llama-server -np for concurrent clients).
• The letters are read from the top-20 log-probabilities at the answer position (a letter outside them counts as probability 0).
• CALIBRATION.json holds the bf16 model's temperatures and thresholds, not validated for these quantisations: refit the thresholds on your own labelled requests before automating decisions.
• No evidence spans; context 4,096 tokens by default (num_ctx in the Modelfile, -c for llama-server).
• Ollama 0.12.11 or newer (measured with 0.35.0); llama.cpp measured with 0.5.0 (b11146).

License

Apache-2.0, like basal-1.5-4.5B.