basal-1.5-4.5B-NVFP4
NVFP4 (NVIDIA Model Optimizer) checkpoint of basal-1.5-4.5B (4.5B) for vLLM on NVIDIA
GPUs. Same model, same prompt format and the same typed answers (choice, noul, score) with a probability for every
option; see the main card for the description, benchmarks, limitations and license.
⚠ Quality warning. The NVFP4 checkpoint of basal-1.5 (4.5B) loses about 2.8 accuracy points (0.902 vs 0.930; agreement 0.937); use
-FP8unless memory or throughput matters more.
Agreement with the bf16 model (fp32 reference): 0.937; accuracy 0.902 (−0.028; bf16 0.930) (1,000 development decisions, both option orders, vLLM 0.30.0 on one RTX PRO 6000 Blackwell; reference: the bf16 weights in fp32). Speed: 17.4 ms per decision at batch 1, 163.4 decisions/s batched, on one RTX PRO 6000 Blackwell (vLLM 0.30.0). Size: 2.9 GB.
What it is
Post-training quantisation with NVIDIA Model Optimizer (NVFP4_DEFAULT_CFG), calibrated on
basal-1.5 calibration decisions: NVFP4 (4-bit floating point E2M1, group 16, FP8 scales) weights and activations in the decoder layers. The token embeddings and the output head stay in bf16, because the decision
is read from the output head's logits. It is a standard ModelOpt Hugging Face export.
Hardware: NVIDIA Blackwell only (B200, B300, RTX 50xx, RTX PRO 6000, DGX Spark).
Run
vLLM brings its own torch, so use a separate environment:
uv pip install "basal[vllm] @ https://github.com/rkinas/basal/archive/refs/tags/v1.5.0.tar.gz"
basal-serve --model Remek/basal-1.5-4.5B-NVFP4 --mode vllm --port 8000
--mode vllm starts with a 4,096-token context and refuses longer states; raise it with --max-len (e.g. 32768 for long
documents). The request and response are the same as for the bf16 model. The basal engine's own modes do not load this
checkpoint; for on-the-fly FP8 in the basal engine, use the bf16 repository with --mode fp8. Evidence spans need the
basal engine.
Calibration
CALIBRATION.json carries the temperatures and confidence thresholds of the bf16 model. The thresholds are not validated
for this checkpoint: refit them on your own labelled requests before automating decisions with them.
License
Apache-2.0, like basal-1.5-4.5B.