Shisa DE-2 FP8-Dynamic
An FP8-dynamic quantization of shisa-ai/shisa-de-2,
at 27.2 GB against 49 GB for the original. The readout contract, inference policy, usage and
benchmarks are the same as the original model's; see its card.
Quantization
| Scheme | FP8_DYNAMIC: 8-bit float weights with per-channel scales, dynamic per-token activation scales |
| Method | Data-free, with llm-compressor 0.13.0 model_free_ptq |
| Quantized | The Linear layers inside the transformer blocks |
| Kept in BF16 | Vision tower, audio tower, token embeddings, output head, MoE router, normalization layers |
recipe.yaml has the equivalent recipe.
Readout test
The 231 public JevBench items, read through the Shisa DE SDK from vLLM 0.30.0 on one RTX PRO 6000, raw probabilities, against the original model on the same server setup:
| Original (BF16) | FP8-dynamic | |
|---|---|---|
| Single read, easy / medium / hard | 48 / 72 / 83 | 48 / 72 / 84 |
| Adaptive policy, easy / medium / hard | 48 / 72 / 86 | 48 / 72 / 85 |
| Same answer as the original, single read | 224 of 231 | |
| Same answer as the original, adaptive policy | 224 of 231 | |
| Mean change in top probability, single read | 0.026 |
Serving
vllm serve shisa-ai/shisa-de-2-FP8-dynamic
from shisa_de import DecisionModel
de = DecisionModel.from_endpoint(
"http://127.0.0.1:8000/v1",
model="shisa-ai/shisa-de-2-FP8-dynamic",
tokenizer="shisa-ai/shisa-de-2", # also applies the original model's calibration
api_key="",
)
de.classify({"ticket": "I was charged twice."}, {"intent": ["refund", "cancel"]})