basal-omni-1.0-ultra-GGUF (experimental)

A GGUF file (Q8_0) of basal-omni-1.0-ultra (31B, base Gemma 4 31B) for llama.cpp and Ollama, on NVIDIA GPUs, Apple Silicon (Metal) or the CPU. Same model and the same typed-decision API through basal-serve. Text only: the image encoder is not in the GGUF file; for images use the bf16 model.

Q8_0 passes every bar of the basal port gate against bf16 (agreement, agreement where bf16 is sure, accuracy, ECE). Q6_K (25.2 GB) is not shipped: it passes the tier's bars, but where bf16 is sure (margin >= 0.2) it agrees in 0.9973 of all decisions, below the 0.998 that a shipped port needs. Smaller quantisations were not built (on mini and max, Q5_K_M and Q4_K_M did not pass).

Measured against bf16

Agreement is the share of decisions where a file picks the same option as the bf16 release model, served as released (--mode eager, bf16), on the development set dev v2 (10,606 decisions), one H100 80 GB (CUDA), the engine's --mode gguf (llama.cpp b11312 through llama-cpp-python 0.3.36, all layers on the GPU). Bars (the basal port gate, as for max): agreement on 2-10 options >= 0.9552, agreement where bf16's margin >= 0.2 >= 0.9893, accuracy within 1 point, ECE at most +0.01. Speed: basal-loadtest, states of about 1,500 tokens with one question, p50 latency one request at a time and decisions/s with 16 in flight.

• Over all 10,606 decisions, where bf16's margin is >= 0.2 Q8_0 agrees in 0.9983 (bar 0.998; Q6_K 0.9973).
• Automation with this folder's CALIBRATION.gguf.json (fitted on Q8_0 readouts with --mode gguf, calibration test of 8,560 decisions, accuracy 0.936): at 1% target error 56.6% of decisions are decided automatically (observed error 0.6%); at 5%, 93.3% (3.8%) (the bf16 model's own calibration: 56.8% and 94.2%).
• The file is a decision-layer port of the same model, not a new model; its benchmark results (Werdykt v1 0.879, the Polish and English sets) are on basal-omni-1.0-ultra. The file is not scored separately on them.

Use

The basal server renders basal's own prompt and reads the probabilities of the option codes; it runs the GGUF file in process, or in front of Ollama or llama.cpp. Install the basal engine 2.3.1 or newer with the gguf extra (llama-cpp-python 0.3.36 or newer: Gemma 4 support):

uv pip install "basal[gguf] @ https://github.com/rkinas/basal/archive/refs/tags/v2.5.3.tar.gz"
hf download Remek/basal-omni-1.0-ultra-GGUF --local-dir basal-omni-1.0-ultra-GGUF
basal-serve --model ./basal-omni-1.0-ultra-GGUF --mode gguf --gguf basal-omni-1.0-ultra-GGUF/basal-omni-1.0-ultra-Q8_0.gguf --port 8000

On NVIDIA, build llama-cpp-python with CUDA (see the engine's docs/GGUF.md); on a Mac it uses Metal. Then ask questions exactly as with basal-omni-1.0-ultra (choice, noul, score, multi, act, tool, and the task types; up to 255 options). The in-process context holds 16,384 tokens.

Ollama (0.35 or newer; to try: checked on the mini files, not measured on ultra):

cd basal-omni-1.0-ultra-GGUF && ollama create basal-omni-1.0-ultra:q8_0 -f Modelfile.Q8_0 && cd ..
basal-serve --model ./basal-omni-1.0-ultra-GGUF --mode ollama --ollama-model basal-omni-1.0-ultra:q8_0 --port 8000

A plain ollama run gives free text only: the typed answers come from basal-serve in front of it. Ollama mode reads at most 20 options per question (Ollama returns at most 20 log-probabilities) and 4,096 tokens per prompt; for long documents and up to 255 options use --mode gguf.

• Calibration: --mode gguf loads CALIBRATION.gguf.json, fitted on this port. CALIBRATION.json is the bf16 model's calibration, the default for --mode ollama and --mode llamacpp until they have fits of their own. You can refit any of them on your own labelled data with basal-calibrate.
• Memory: the weights take 32.6 GB, plus the context; on the H100 the server peaked at 49 GB of GPU memory in the speed test. Not measured on Apple Silicon; on a Mac, plan for 64 GB of memory.
• One option order. The shipped calibration is fitted for one option order (the default).

How it was made

• convert_hf_to_gguf.py of llama.cpp b11312 on the bf16 release model (the text decoder; the image encoder is left out) to Q8_0.
• CALIBRATION.gguf.json: the bf16 model's method and splits on this port's Q8_0 readouts (--mode gguf, one option order): per-type and per-bucket temperatures and certified thresholds for 1% and 5% error.

Files

• basal-omni-1.0-ultra-Q8_0.gguf: the GGUF file.
• Modelfile.Q8_0: for ollama create.
• Tokenizer, chat_template.jinja: as in the bf16 model (the basal server renders the prompt with them).
• CALIBRATION.gguf.json, CALIBRATION.json: the calibration fitted on this port, and the bf16 model's.
• basal.json: prompt format 2 and the readout protocol.
• LM Studio: basal-omni-1.0-ultra-Q8_0-decision.gguf is the file above with basal's decision metadata added (weights unchanged). LM Studio 0.4.26 or newer (beta channel at the time of writing; llama.cpp engine 2.65.1+, not the MLX engine) serves it as a decision model on /v1/systemone and /v1/decisions: choice, yes/no and score, up to 26 options, temperatures from CALIBRATION.gguf.json. LM Studio ends a request after 30 s, so split very long documents.
  • With basal 2.5.0 you can add the same metadata to a GGUF file yourself (a file of this repository, downloaded, or your own quantisation; the tensors are not touched): basal-export-gguf Remek/basal-omni-1.0-ultra-GGUF lmstudio/basal-omni-1.0-ultra-Q8_0.gguf --gguf basal-omni-1.0-ultra-Q8_0.gguf --systemone.

Limitations

• Experimental, as the bf16 model: validate it on your own documents before relying on it.
• Text only (see above).
• Quantisation changes some decisions where the model is unsure (see the table); decisions it is sure about are almost unchanged.
• Exact arithmetic and thresholds are a weak spot of one-pass decisions: compute them in code.
• Decisions with serious consequences for people should be reviewed by a person.

Citation

@software{kinas2026basal20,
  title   = {basal-2.0: typed-decision models and inference engine for Polish and English},
  author  = {Kinas, Remigiusz},
  year    = {2026},
  version = {2.0.0},
  url     = {https://github.com/rkinas/basal}
}

License and attribution

Apache-2.0. Converted from basal-omni-1.0-ultra, fine-tuned from google/gemma-4-31B-it (Apache-2.0).