Shisa DE-1 FP8-Dynamic

This is an FP8-dynamic (W8A8) quantization of shisa-ai/shisa-de-1, the general-purpose decision model trained from google/gemma-4-26B-A4B-it.

The readout contract is unchanged. The model still answers one question per request by reading the next-token distribution at the answer boundary and restricting it to the option letters. The base model card is authoritative for that contract, the serving scaffold, and the task-quality tables; this card covers what quantization changed and what it did not.

Quantization

The checkpoint is structurally identical to RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic — 24,198 tensors against their 24,199 — and was built with the same scheme and the same ignore policy. The single difference is lm_head.weight, which their checkpoint materializes and this one does not, because the BF16 source leaves it tied to embed_tokens.

FP8_BLOCK (128×128) is not usable for this architecture. The compressed-tensors FP8 MoE path requires intermediate_size_per_partition % 128 == 0 and moe_intermediate_size is 704; the dense MLP is misaligned too (intermediate_size 2112); and the MoE path hardcodes kFp8Static128BlockSym, so a smaller block does not help.

Readout fidelity

The quantization was gated on the readout it is deployed through. 65 questions spanning 15 families and 2 to 26 options were replayed through both arms on the identical scaffold, with max_tokens=1, temperature=0, and the option-letter rows read from logprobs:

No argmax moved and no near-tie margin collapsed. The largest shift is on one deliberately ambiguous question where BF16 splits 0.555/0.432 and FP8 sharpens to 0.746/0.242 — the same argmax, on the question with the smallest BF16 margin in the set. Mean margin moved up rather than down, which is the shape of a healthy quantization rather than a lossy one. The fixture carries the base card's worked example verbatim and both arms reproduce its argmax and near-1.0 A probability.

The gate is measured on a quiesced service. vLLM batches whatever is queued at each scheduler step, so a serial request can still be co-batched with unrelated traffic, which changes kernel selection and the resulting reduction order. A single co-batched request is enough to shift raw letter logprobs by up to 0.795 — not jitter, since all loaded responses are identical to each other and the shift reverses when load stops. The option-letter argmax does not move, but logprob-derived quantities (the restricted softmax, the margin, top-k membership) are exposed. Quiesce, wait a second, then measure.

Serving

The measured working stack is vLLM 0.25.1 with transformers 5.6.2 and torch 2.11.0+cu130:

vllm serve shisa-ai/shisa-de-1-FP8-dynamic \
  --max-model-len 4096 \
  --moe-backend triton

Two constraints matter:

• vLLM 0.25 or newer. model_free_ptq writes per-expert 2-D experts (layers.N.experts.{i}.{gate,up,down}_proj.weight). vLLM 0.19.0 cannot load that layout and fails with a KeyError; vLLM ≥ 0.25 remaps .experts.{i}. to .moe.experts.{i}..
• transformers older than 5.15. transformers 5.15 moved global_head_dim into the per-layer heterogeneity system, so vLLM's getattr(config, "global_head_dim", config.head_dim) resolves to head_dim 256 on the full-attention layers and the k_norm load fails (256 ← 512). The same change is why the config this artifact was quantized from is not loadable by older transformers without normalization.

The checkpoint is auto-detected as quantization=compressed-tensors. It uses CutlassFP8ScaledMMLinearKernel for the linear layers and the TRITON FP8 MoE backend. Force --moe-backend triton on hosts without a CUDA toolkit: the auto-selected FlashInfer CUTLASS MoE path JIT-compiles at first inference and fails when nvcc is absent. VLLM_USE_FLASHINFER_SAMPLER=0 avoids the same failure in the sampler.

Because the artifact does not materialize lm_head.weight, serving must keep tie_word_embeddings=true. Do not override it.

Measured serving performance

Measured on a reference deployment of two RTX 5090s (32,607 MiB nominal each) at TP2, serving --max-model-len 32768 --max-num-seqs 64 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.90 --moe-backend triton --enable-prefix-caching --enforce-eager. Sustained 20 s per level, short prompt, max_tokens=1:

Memory is set at startup and does not move under load: 31,404 MiB per card, flat across the ramp, across six and eight concurrent 32,760-token prompts, and across a single 32,767-token request — 13,460 requests, zero errors, 708 MiB free. If the service starts on a 32 GiB card, this traffic shape will not push it over.

Two sizing facts that are not obvious from the flags:

• gpu_memory_utilization 0.90 bounds torch allocations, not the process. vLLM's own accounting sums to 28.22 GiB, but nvidia-smi attributes 31,394 MiB to the worker: roughly 2.5 GiB of CUDA context, NCCL/TP buffers, and allocator overhead sits outside the budget. True card occupancy is 96.3%. On a card with less headroom than the 5090, a service that "fits at 0.90" can still fail to start.
• --max-num-batched-tokens binds long prompts, not KV. At 32,768 tokens the KV cache holds 6.70 concurrent requests, but the 8192-token prefill budget lets roughly one long request make progress at a time. Six and eight concurrent 32,760-token prompts queue rather than fill the cache, at 18,055 and 18,129 prefill tok/s respectively.

Full method, raw logs, and the qualification of the physical 32 GiB budget are in gemma4/evidence/de1-serving-envelope-20260929/ in the lane repository.

Files

Limitations

• Task accuracy was not re-measured. The 65-question probe is sized to detect a collapsed readout, not to re-score the base model's 29-suite harness. Treat the table above as a regression gate, not as an accuracy claim.
• The serving numbers are one deployment's. They come from a TP2 pair of RTX 5090s, and do not transfer to other hardware, other TP degrees, or other --max-num-seqs settings. Treat them as an envelope qualification, not as a portable benchmark.
• The probe is text-only. The vision and audio towers are untouched by quantization but were not exercised.
• FP8_DYNAMIC is not a hardware-portable format. It requires FP8 support and the compressed-tensors kernel paths; it is not a CPU or older-GPU checkpoint.

Provenance

Quantized by Shisa AI from shisa-ai/shisa-de-1 at revision 9111f4746a96d6cdf9f1a23a88afd5b57244508b. The recipe, the ignore policy, and the verification checks are recorded in the base model card's lane documentation at shisa-ai/sm120-tuning (SHISA-DE-1.md), together with the readout evidence and the quantizer source.

Licensed under Apache-2.0, inheriting the Gemma 4 license terms of the base model.