basal-omni-1.0-ultra-FP8 (experimental)

FP8 checkpoint of basal-omni-1.0-ultra (31B, base Gemma 4 31B), served through vLLM. Same model and the same typed-decision API through basal-serve, at 33 GB instead of 59 GB: it fits one 80 GB or 96 GB GPU with room for long states and parallel requests. Text only: for images, use the bf16 model.

Passes every bar of the basal port gate against bf16 (agreement, agreement where bf16 is sure, accuracy, ECE), and its own calibration automates as many decisions at 1% error as bf16.

Measured against bf16

Agreement is the share of decisions where this checkpoint picks the same option as the bf16 release model, served as released (--mode eager, bf16, one H100), on the development set dev v2 (10,606 decisions). Bars (the basal-2.0 port gate): agreement on 2-10 options >= 0.9552, agreement where bf16's margin >= 0.2 >= 0.9893, accuracy within 1 point, ECE at most +0.01.

• The RTX PRO 6000 rows are compared with the same bf16 rows (H100), so the difference between the GPUs is inside its numbers.
• All 10,606 decisions (up to 255 options), H100: the same option in 97.5%, accuracy 0.8749 -> 0.8701 (-0.48 pt). Most changed decisions are near-ties of bf16 (margin below 0.05: 56% agreement; margin 0.5 or more: 99.7%).
• Automation (calibration test, 8,560 decisions, shipped thresholds of each package): at 1% target error this checkpoint decides 56.9% of decisions automatically (bf16: 56.8%); at 5%, 92.8% (bf16: 94.2%).
• Measured with basal 2.0. basal 2.1 only adds task types: one request at a time it returns the same probabilities as 2.0, bit for bit (2,000 dev v2 decisions, H100). With 32 requests in flight, vLLM's batching changes the top answer between any two runs on about 1% of decisions (0.3% where the margin is 0.2 or more).
• The checkpoint is a decision-layer port of the same model, not a new model; its benchmark results (Werdykt v1 0.879, the Polish and English sets) are on basal-omni-1.0-ultra. This port is not scored separately on them.

Use

Install the basal engine v2.5.0 with vLLM 0.31.0 (its PyPI build uses CUDA 13: NVIDIA driver 580 or newer):

uv venv --python 3.12 --allow-existing ~/basal-vllm && source ~/basal-vllm/bin/activate
uv pip install vllm==0.31.0 "basal @ https://github.com/rkinas/basal/archive/refs/tags/v2.5.3.tar.gz"
basal-serve --model Remek/basal-omni-1.0-ultra-FP8 --mode vllm --max-len 16384 --gpu-memory 0.9 --port 8000

Then ask questions exactly as with basal-omni-1.0-ultra (choice, noul, score, multi, act, tool; up to 255 options).

• GPUs: FP8 needs sm89 or newer (Ada, Hopper, Blackwell). Checked on one H100 80 GB and one RTX PRO 6000 96 GB.
• Memory: the weights take 33 GB. With --gpu-memory 0.9 on an 80 GB H100, vLLM keeps 38,360 tokens of state cache (2.3 states of 16,384 tokens at once); with the bf16 model it keeps 8,348, too few for one 16,384-token state. On a 96 GB RTX PRO 6000: 55,279 tokens.
• basal 2.5 also serves it with --mode fast or --mode fp8 on one 80 GB H100 (the checkpoint loads without a bf16 copy; 44.8 GB after start-up).
• Text only. --mode vllm reads text states; images ("media") need the bf16 model with --mode eager --media on.
• One option order. The shipped calibration is fitted for one option order (the default).
• vLLM batches requests its own way, so a decision near a tie can change with the load (--deterministic is not available in --mode vllm).

Speed

One NVIDIA H100 80 GB, one option order, 16 requests in flight (basal-loadtest); latency p50 / p95 is one request at a time:

How it was made

• Recipe: NVIDIA ModelOpt 0.47, FP8 (E4M3) weights and activations with static per-tensor scales, calibrated on 2,048 Polish and English decision prompts in the model's prompt format; the first two and last two decoder layers, the embeddings, the output head and the image encoder stay bf16 (383 of the language model's linear layers are FP8).
• Calibration: CALIBRATION.vllm.json (and its copy CALIBRATION.json) is fitted on this checkpoint's own vLLM readouts with the bf16 model's method and splits: per-type temperatures and certified thresholds for 1% and 5% error, one option order. basal-serve --mode vllm loads CALIBRATION.vllm.json.

Files

• model-*.safetensors, config.json, hf_quant_config.json: the ModelOpt FP8 checkpoint (vLLM reads it natively).
• Tokenizer, chat_template.jinja, processor_config.json: as in the bf16 model.
• CALIBRATION.vllm.json, CALIBRATION.json: the calibration fitted on this checkpoint.
• basal.json: prompt format 2 and the readout protocol.

Limitations

• Experimental, as the bf16 model: validate it on your own documents before relying on it.
• Text only in this checkpoint (see Use); the 31B base has no audio encoder.
• FP8 changes some decisions where the model is unsure (see the table); decisions it is sure about are almost unchanged.
• Exact arithmetic and thresholds are a weak spot of one-pass decisions: compute them in code.
• Decisions with serious consequences for people should be reviewed by a person.

Citation

@software{kinas2026basal20,
  title   = {basal-2.0: typed-decision models and inference engine for Polish and English},
  author  = {Kinas, Remigiusz},
  year    = {2026},
  version = {2.0.0},
  url     = {https://github.com/rkinas/basal}
}

License and attribution

Apache-2.0. Quantised from basal-omni-1.0-ultra, fine-tuned from google/gemma-4-31B-it (Apache-2.0).