Imajev-4B FP8 SM120

The tuned profile targets an RTX PRO 6000 Blackwell (SM120). Its full DecisionBench results compare this FP8 SM120 profile directly with the BF16 original.

DecisionBench

Both runs completed all 23,900 rows with no failures, exclusions, or truncation. FP8 SM120's mean request latency was 36.4% lower at concurrency 1. ECE is nearly identical; the BF16 original had slightly better mean NLL. The two profiles selected different answers on 324 rows; BF16 was correct on 148 rows where FP8 SM120 was not, and FP8 SM120 was correct on 125 where BF16 was not.

The suite used four option rotations and one request at a time. These are end-to-end HTTP latencies, not concurrent throughput. The full report includes matched row and request verification, per-task results, and latency distributions in benchmarks/decisionbench-full.json.

The top-choice accuracy and aggregate calibration metrics do not describe every probability equally closely. Against BF16, candidate probabilities had median absolute difference 0.00000462, p99 0.02511, and maximum 0.36081. In 218/23,900 rows (0.91%), at least one candidate probability differed by 0.10 or more. Probability-difference counts are in the full report.

GPU Support

This package is optimized and benchmarked on one RTX PRO 6000 Blackwell (SM120, 96 GB). NVIDIA's compute_120f family target covers compute capability 12.0 and 12.1, so the kernels are expected to be portable and perform well on SM121 too. SM121 runtime performance has not been measured here. See NVIDIA's compute-capability compatibility table.

The measured process-memory peak was 19,968 MiB, sampled once per second during warmup and the full suite. The GPU power limit was 395 W and was not changed by the launcher. Allow at least 32 GiB of free VRAM for a comfortable margin; larger image inputs may need more.

What It Contains

• Qwen3.5-4B converted from the pinned BF16 weights to dynamic E4M3 FP8, with 128x128 weight blocks and FP32 scales.
• Imajev's trained LoRA adapter, decision readout, and calibration. Adapter tensors are distributed unchanged in FP32; the SM120 runtime computes LoRA in BF16 and keeps the readout in FP32.
• BF16 vision, embeddings, norms, and excluded projection layers. This is mixed-precision inference, not an all-FP8 model.
• The SM120 CUDA runtime, vision-feature reuse, model conversion code, checksums, pinned dependencies, and full evaluation receipts.

No new model training or calibration fitting was performed. The adapter is not merged into the FP8 base. This package is independent of the original model authors.

Quick Start

Use Docker, NVIDIA Container Toolkit, and a driver compatible with CUDA 13.0. Python 3.11 or newer is needed for the included verification and client scripts.

Download the repository, including adapter/ and runtime/:

hf download zenmagnets/Imajev-4B-FP8-SM120 --local-dir ./imajev-4b-fp8-sm120
cd imajev-4b-fp8-sm120
python3 runtime/verify_release.py .
docker build -t imajev-fp8-sm120:local .
docker run --rm --gpus device=0 --cpus 12 --shm-size 4g \
  -p 127.0.0.1:8765:8765 \
  -v "$PWD:/models:ro" \
  -v imajev-fp8-cache:/root/.cache \
  imajev-fp8-sm120:local

The first startup downloads a pinned Hugging Face kernel into the cache and compiles GPU kernels. Once that cache is populated, add -e HF_HUB_OFFLINE=1 to run offline. The model itself is loaded from the mounted local repository; no Hugging Face token is needed after download.

The optimized worker passed a native DecisionBench smoke, the packaged functional suite including vision, and the full DecisionBench run. See benchmarks/validation.json and benchmarks/release-validation.json for the checks and runtime versions.

Readiness is GET /v1/models; decisions use POST /v1/systemone. This typed-decision API does not expose /v1/chat/completions.

curl -s http://localhost:8765/v1/models
curl -s http://localhost:8765/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "state": {"ticket": "The same invoice was charged twice. Please refund the duplicate."},
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which department should handle this ticket?",
        "criteria": {
          "billing": "Payments and refunds",
          "technical_support": "Product failures",
          "sales": "New purchases"
        }
      }
    }
  }'

Answers contain the selected value, probabilities, unknown_probability, and abstained. Supported question types are choice, noul, and score. Up to two images can be supplied as data URLs in the images list. See Imajev's API specification for the complete contract.

The included Python client sends text, one-image, or two-image requests:

python3 examples/client.py --base-url http://localhost:8765
python3 examples/client.py --base-url http://localhost:8765 --image ./product.jpg
python3 examples/client.py --base-url http://localhost:8765 --image ./first.jpg --image ./second.jpg

The upstream server handles GPU requests serially; simultaneous callers queue. The configured input limit is 65,536 tokens. CUDA graphs cover inputs up to 4,096 tokens; the longest request in the evaluation contained 39,111 tokens.

SM120 Runtime

The tuned path combines FlashInfer's SM120 CUTLASS block-FP8 GEMM, a fused Triton activation quantizer, activation-quantization reuse within a decoder layer, BF16 LoRA arithmetic, FP32 decision readout, and vision-feature reuse across option rotations and questions.

The original MTP weights are preserved in the checkpoint but are not loaded or used by the decision-serving path.

Provenance

The base, Imajev code, and conversion recipe are Apache-2.0 licensed. See LICENSE, NOTICE, and MODEL_PROVENANCE.json. The adapter configuration's original training-machine path was replaced with its public base-model ID and pinned revision; the adapter, readout, and calibration tensor bytes are unchanged.