basal-omni-1.0-mini-GGUF (experimental)

GGUF files (Q8_0 and Q6_K) of basal-omni-1.0-mini (E4B, base Gemma 4 E4B) for llama.cpp and Ollama, on NVIDIA GPUs, Apple Silicon (Metal) or the CPU. Same model and the same typed-decision API through basal-serve. Text only: the image and audio encoders are not in the GGUF files; for images and audio use the bf16 model.

Q8_0 and Q6_K pass every bar of the basal port gate against bf16 (agreement, agreement where bf16 is sure, accuracy, ECE). Q5_K_M and Q4_K_M did not, so they are not shipped.

Measured against bf16

Agreement is the share of decisions where the file picks the same option as the bf16 release model, served as released (--mode eager, bf16), on the development set dev v2 (10,606 decisions), one RTX PRO 6000 96 GB (CUDA), the engine's --mode gguf (llama.cpp b11312 through llama-cpp-python 0.3.36). Bars (the basal port gate for this tier): agreement on 2-10 options >= 0.9161, agreement where bf16's margin >= 0.2 >= 0.9858, accuracy within 1 point, ECE at most +0.01.

• For reference, an F16 conversion (14.9 GB, not shipped) agrees in 0.9949 (0.9999 at margin >= 0.2): Gemma 4 is implemented faithfully by llama.cpp, and llama.cpp tokenizes the basal prompts exactly as the Hugging Face tokenizer (2,000 of 2,000).
• Q8_0 on Metal (Apple M4 Pro, 2,000 decisions): agreement 0.9887, 0.9992 at margin >= 0.2, accuracy +0.13 pt; 1.2 decisions/s in process. Ollama 0.35.0 (Q8_0, 300 decisions with up to 10 options, one at a time): agreement 0.9800, 0.9961 at margin >= 0.2, accuracy +0.33 pt, 2.1 decisions/s.
• Automation with this folder's CALIBRATION.gguf.json (fitted on Q8_0 readouts with --mode gguf, calibration test of 8,560 decisions, accuracy 0.882): at 1% target error 42.4% of decisions are decided automatically (observed error 0.7%); at 5%, 78.3% (3.9%).
• The files are a decision-layer port of the same model, not a new model; its benchmark results (Werdykt v1 0.769, the Polish and English sets) are on basal-omni-1.0-mini. These files are not scored separately on them.

Use

The basal server renders basal's own prompt and reads the probabilities of the option codes; it runs the GGUF file in process, or in front of Ollama or llama.cpp. Install the basal engine 2.2.0 or newer with the gguf extra (llama-cpp-python 0.3.36 or newer: Gemma 4 support):

uv pip install "basal[gguf] @ https://github.com/rkinas/basal/archive/refs/tags/v2.5.3.tar.gz"
hf download Remek/basal-omni-1.0-mini-GGUF --local-dir basal-omni-1.0-mini-GGUF
basal-serve --model ./basal-omni-1.0-mini-GGUF --mode gguf --gguf basal-omni-1.0-mini-GGUF/basal-omni-1.0-mini-Q8_0.gguf --port 8000

On NVIDIA, build llama-cpp-python with CUDA (see the engine's docs/GGUF.md); on a Mac it uses Metal. Then ask questions exactly as with basal-omni-1.0-mini (choice, noul, score, multi, act, tool, and the task types; up to 255 options).

Ollama (0.35 or newer; checked on mini):

cd basal-omni-1.0-mini-GGUF && ollama create basal-omni-1.0-mini:q8_0 -f Modelfile.Q8_0 && cd ..
basal-serve --model ./basal-omni-1.0-mini-GGUF --mode ollama --ollama-model basal-omni-1.0-mini:q8_0 --port 8000

A plain ollama run gives free text only: the typed answers come from basal-serve in front of it. Ollama mode reads at most 20 options per question (Ollama returns at most 20 log-probabilities) and 4,096 tokens per prompt; for long documents and up to 255 options use --mode gguf.

• Calibration: --mode gguf loads CALIBRATION.gguf.json, fitted on this port. CALIBRATION.json is the bf16 model's calibration, the default for --mode ollama and --mode llamacpp until they have fits of their own. You can refit any of them on your own labelled data with basal-calibrate.
• Memory on Apple Silicon: mini Q8_0 runs on a 16 GB Mac; long states need more memory.
• One option order. The shipped calibration is fitted for one option order (the default).

How it was made

• convert_hf_to_gguf.py of llama.cpp b11312 on the bf16 release model (the text decoder; the image and audio encoders are left out) to F16 and Q8_0; llama-quantize from F16 to Q6_K without an importance matrix. Converting on Linux (CUDA) and macOS gives byte-identical files.
• CALIBRATION.gguf.json: the bf16 model's method and splits on this port's Q8_0 readouts (--mode gguf, one option order): per-type and per-bucket temperatures and certified thresholds for 1% and 5% error. Q6_K is served with the same file.

Files

• basal-omni-1.0-mini-Q8_0.gguf, basal-omni-1.0-mini-Q6_K.gguf: the GGUF files.
• Modelfile.Q8_0, Modelfile.Q6_K: for ollama create.
• Tokenizer, chat_template.jinja: as in the bf16 model (the basal server renders the prompt with them).
• CALIBRATION.gguf.json, CALIBRATION.json: the calibration fitted on this port, and the bf16 model's.
• basal.json: prompt format 2 and the readout protocol.
• LM Studio: basal-omni-1.0-mini-Q8_0-decision.gguf and basal-omni-1.0-mini-Q6_K-decision.gguf are the files above with basal's decision metadata added (weights unchanged). LM Studio 0.4.26 or newer (beta channel at the time of writing; llama.cpp engine 2.65.1+, not the MLX engine) serves them as decision models on /v1/systemone and /v1/decisions: choice, yes/no and score, up to 26 options, temperatures from CALIBRATION.gguf.json. LM Studio ends a request after 30 s, so split very long documents.
  • With basal 2.5.0 you can add the same metadata to a GGUF file yourself (a file of this repository, downloaded, or your own quantisation; the tensors are not touched): basal-export-gguf Remek/basal-omni-1.0-mini-GGUF lmstudio/basal-omni-1.0-mini-Q8_0.gguf --gguf basal-omni-1.0-mini-Q8_0.gguf --systemone.

Limitations

• Experimental, as the bf16 model: validate it on your own documents before relying on it.
• Text only (see above).
• Quantisation changes some decisions where the model is unsure (see the table); decisions it is sure about are almost unchanged.
• Exact arithmetic and thresholds are a weak spot of one-pass decisions: compute them in code.
• Decisions with serious consequences for people should be reviewed by a person.

Citation

@software{kinas2026basal20,
  title   = {basal-2.0: typed-decision models and inference engine for Polish and English},
  author  = {Kinas, Remigiusz},
  year    = {2026},
  version = {2.0.0},
  url     = {https://github.com/rkinas/basal}
}

License and attribution

Apache-2.0. Converted from basal-omni-1.0-mini, fine-tuned from google/gemma-4-E4B-it (Apache-2.0).