Shisa DE-2.1 FP8-Dynamic
An FP8-dynamic quantization of shisa-ai/shisa-de-2.1,
at 27.2 GB against 51.6 GB for the original. The readout contract, inference policy, usage and
benchmarks are the same as the original model's; see its card.
Quantization
| Scheme | FP8_DYNAMIC: 8-bit float weights with per-channel scales, dynamic per-token activation scales |
| Method | Data-free, with llm-compressor 0.13.0 model_free_ptq |
| Quantized | The Linear layers inside the transformer blocks |
| Kept in BF16 | Vision tower, audio tower, token embeddings, output head, MoE router, normalization layers |
recipe.yaml has the equivalent recipe.
Readout test
The 231 public JevBench items, read through the Shisa DE SDK from vLLM 0.30.0 on one RTX PRO 6000, raw probabilities, against the original model on the same server setup:
| Original (BF16) | FP8-dynamic | |
|---|---|---|
| Single read, easy / medium / hard | 48 / 72 / 85 | 48 / 72 / 84 |
| Adaptive policy, easy / medium / hard | 48 / 72 / 91 | 48 / 72 / 93 |
| Same answer as the original, single read | 224 of 231 | |
| Same answer as the original, adaptive policy | 221 of 231 | |
| Mean change in top probability, single read | 0.017 |
Serving
vllm serve shisa-ai/shisa-de-2.1-FP8-dynamic
from shisa_de import DecisionModel
de = DecisionModel.from_endpoint(
"http://127.0.0.1:8000/v1",
model="shisa-ai/shisa-de-2.1-FP8-dynamic",
api_key="",
)
de.classify({"ticket": "I was charged twice."}, {"intent": ["refund", "cancel"]})
shisa-de 0.6.1 and later apply the original model's calibration to this repo by its name. With
0.6.0, pass tokenizer="shisa-ai/shisa-de-2.1" to do the same.