basal-omni-1.0-mini-FP8 (experimental)

FP8 checkpoint of basal-omni-1.0-mini (E4B, base Gemma 4 E4B), served through vLLM. Same model and the same typed-decision API through basal-serve, at 12 GB of weights instead of 16 GB, and faster (see Speed). Text only: for images and audio, use the bf16 model.

Passes every bar of the basal port gate against bf16 (agreement, agreement where bf16 is sure, accuracy, ECE).

Measured against bf16

Agreement is the share of decisions where this checkpoint picks the same option as the bf16 release model, served as released (--mode eager, bf16), on the development set dev v2 (10,606 decisions), both on one RTX PRO 6000 96 GB (Blackwell). Bars (the basal port gate for this tier): agreement on 2-10 options >= 0.9161, agreement where bf16's margin >= 0.2 >= 0.9858, accuracy within 1 point, ECE at most +0.01.

• All 10,606 decisions (up to 255 options): the same option in 95.6%, accuracy 0.7560 -> 0.7530 (-0.30 pt). Most changed decisions are near-ties of bf16 (margin below 0.05: 66% agreement; margin 0.5 or more: 99.6%).
• Automation (calibration test, 8,560 decisions, each package with thresholds fitted on its own readouts on the same GPU): at 1% target error this checkpoint decides 41.9% of decisions automatically (bf16: 41.7%); at 5%, 63.5% (bf16: 77.6%). At 5% its choice threshold is held at the 1% one (a looser target never gets a stricter threshold), so it automates less there than bf16 does.
• Repeatability. In --mode vllm a decision can change between runs: vLLM's numbers depend on what the server computed before (prefix cache, batching). Two runs of the first 2,000 dev v2 decisions (one at a time, and 8 in flight) give the same top answer in 96.8% of them (99.2% where bf16's margin is 0.2 or more); the released engine v2.1.0 against v2.0.0 differs by the same amount. For repeatable answers set VLLM_BATCH_INVARIANT=1 (see Use): two runs of basal-omni-1.0-max-FP8 at 32 requests in flight then gave the same top answer on all 2,000 decisions, at about 23% fewer decisions per second.
• The checkpoint is a decision-layer port of the same model, not a new model; its benchmark results (Werdykt v1 0.769, the Polish and English sets) are on basal-omni-1.0-mini. This port is not scored separately on them.

Use

Install the basal engine with vLLM 0.31.0 (its PyPI build uses CUDA 13: NVIDIA driver 580 or newer):

uv venv --python 3.12 --allow-existing ~/basal-vllm && source ~/basal-vllm/bin/activate
uv pip install vllm==0.31.0 "basal @ https://github.com/rkinas/basal/archive/refs/tags/v2.2.0.tar.gz"
basal-serve --model Remek/basal-omni-1.0-mini-FP8 --mode vllm --max-len 16384 --gpu-memory 0.9 --port 8000

Then ask questions exactly as with basal-omni-1.0-mini (choice, noul, score, multi, act, tool, and the task types; up to 255 options).

• GPUs: FP8 needs sm89 or newer (Ada, Hopper, Blackwell). Checked on one RTX PRO 6000 96 GB.
• Memory: the weights take 12 GB. With --gpu-memory 0.9 on the RTX PRO 6000, vLLM keeps 1,306,616 tokens of state cache (bf16 through vLLM: 1,244,745).
• Text only. --mode vllm reads text states; images and audio ("media") need the bf16 model with --mode eager --media on.
• One option order. The shipped calibration is fitted for one option order (the default).
• Repeatable answers: start the server with VLLM_BATCH_INVARIANT=1 basal-serve ... (about 23% fewer decisions per second); without it a decision near a tie can change between runs and with the load (--deterministic is not available in --mode vllm).

Speed

One RTX PRO 6000 96 GB, one option order, 16 requests in flight (basal-loadtest); latency p50 / p95 is one request at a time:

How it was made

• Recipe: NVIDIA ModelOpt 0.47, FP8 (E4M3) weights and activations with static per-tensor scales, calibrated on 2,048 Polish and English decision prompts in the model's prompt format; decoder layers 0, 1, 40 and 41, the embeddings, the output head and the image and audio encoders stay bf16 (311 of the language model's linear layers are FP8).
• Calibration: CALIBRATION.vllm.json (and its copy CALIBRATION.json) is fitted on this checkpoint's own vLLM readouts with the bf16 model's method and splits: per-type temperatures and certified thresholds for 1% and 5% error, one option order. basal-serve --mode vllm loads CALIBRATION.vllm.json. You can refit it on your own labelled data with basal-calibrate.

Files

• model-*.safetensors, config.json, hf_quant_config.json: the ModelOpt FP8 checkpoint (vLLM reads it natively).
• Tokenizer, chat_template.jinja, processor_config.json: as in the bf16 model.
• CALIBRATION.vllm.json, CALIBRATION.json: the calibration fitted on this checkpoint.
• basal.json: prompt format 2 and the readout protocol.

Limitations

• Experimental, as the bf16 model: validate it on your own documents before relying on it.
• Text only in this checkpoint (see Use).
• FP8 changes some decisions where the model is unsure (see the table); decisions it is sure about are almost unchanged.
• Exact arithmetic and thresholds are a weak spot of one-pass decisions: compute them in code.
• Decisions with serious consequences for people should be reviewed by a person.

Citation

@software{kinas2026basal20,
  title   = {basal-2.0: typed-decision models and inference engine for Polish and English},
  author  = {Kinas, Remigiusz},
  year    = {2026},
  version = {2.0.0},
  url     = {https://github.com/rkinas/basal}
}

License and attribution

Apache-2.0. Quantised from basal-omni-1.0-mini, fine-tuned from google/gemma-4-E4B-it (Apache-2.0).