NVFP4 / FP8 mixed-precision checkpoint of ScarletKc-2.0, produced by ModelOpt Local Hessian post-training quantization (PTQ) directly from the BF16 weights at revision ddebd3b. The upstream model card is included as SOURCE_MODEL_CARD.md.
| Property | Value |
|---|---|
| Quantization | NVFP4 / FP8 mixed precision (ModelOpt Local Hessian PTQ) |
| Recipe | NVIDIA Qwen3.8-27B recipe nvfp4_w4a4_mlp_fp8_attn_local_hessian: NVFP4 W4A4 for MLP and lm_head, FP8 for attention projections covered by the recipe |
| Quantized modules | 193 NVFP4, 208 FP8 |
| Vision and MTP | Kept in BF16; all 333 vision tensors and 15 MTP tensors match the source by SHA-256 |
| Size | 20.42 GiB in three safetensors shards (BF16 source: 51.75 GiB) |
| Context | 262,144 tokens; original tokenizer and chat template |
| Toolchain | ModelOpt 0.48.0 in the NVIDIA vLLM 26.10 container, single DGX Spark (GB10) |
512 samples from Nemotron datasets, rendered with the original ScarletKc chat template, capped at 2,048 tokens each, batch size 1:
Calibration JSONL SHA-256: 4c96d113e399783deb2b9d3a6256529a4fe46da0478ce4267a6712960c74bd75
Local Hessian uses the calibration input activations to estimate how weight error propagates to each layer's output, and picks the scale of every weight block accordingly. The recipe was validated by NVIDIA on base Qwen3.8-27B; it has not been separately benchmarked against the BF16 ScarletKc-2.0.
Requires an NVFP4-capable Blackwell GPU and vLLM with ModelOpt mixed-precision support (tested with the NVIDIA nvcr.io/nvidia/vllm:26.10-py3 container). The quantization format is detected from hf_quant_config.json.
vllm serve SayakaNeko/ScarletKc-2.0-NVFP4 \
--served-model-name scarletkc \
--max-model-len 262144 \
--gpu-memory-utilization 0.60 \
--max-num-seqs 1 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill \
--no-enable-prefix-caching \
--mamba-cache-mode align \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--enforce-eager \
--kernel-config '{"enable_flashinfer_autotune":false}' \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Drop --speculative-config to run without MTP. Thinking control and sampling defaults are the same as the upstream model (see SOURCE_MODEL_CARD.md).
vLLM 26.10, FlashInfer CUTLASS NVFP4 GEMM, --max-model-len 262144, --gpu-memory-utilization 0.60, single sequence, eager mode.
| Check | MTP off | MTP (3 draft tokens) |
|---|---|---|
| Image input (two-color ordering, 2 prompts) | 2 / 2 | 2 / 2 |
| Decode speed, 192-token outputs, 3 prompts × 2 runs | 11.6 tok/s | 17.9–28.3 tok/s (draft acceptance 32–72%) |
| TTFT, short prompt | 0.16 s | 0.19–0.22 s |
| Single-needle retrieval, 262,112-token prompt, needle at token 131,056 | Pass | Pass |
| Prefill time for the 262,112-token prompt | 386 s | 403 s |
| KV cache capacity | 776,349 tokens | 696,994 tokens |
Earlier 8K-context text checks also covered the ScarletKc identity response in Traditional Chinese, Japanese instruction following and English arithmetic.
These are functional checks, not a quality benchmark. No perplexity or task-accuracy comparison against the BF16 model has been run.
Everything in the upstream card's Limitations section applies, in particular the reduced refusal behavior inherited from JonathanColetti/Qwen3.8-27B-Uncensored. Add external moderation before exposing this model to untrusted users. 4-bit quantization may introduce additional quality loss that the checks above do not measure.
Apache License 2.0, same as ScarletKc-2.0. See LICENSE and NOTICE. Modification in this repository: post-training quantization of the ScarletKc-2.0 BF16 weights to NVFP4 / FP8 with NVIDIA ModelOpt.
model-00002-of-00003.safetensors 10.0 GBmodel-00001-of-00003.safetensors 10.0 GBmodel-00003-of-00003.safetensors 2.0 GBScarletKc/ScarletKc-2.0