Sage-A1.5-Ablitrex (abliterated) — AMD Quark MXFP4

MXFP4 quantized checkpoint of the abliterated Sage-A1.5 model (mostwanted002/Sage-A1.5-Ablitrex-abliterated-BF16), converted to the AMD Quark MXFP4 format (OCP MX: 4-bit E2M1 weights with per-32-group E8M0 scales, dynamic per-32-group MXFP4 activation quantization at runtime).

Contents

• All weights of the abliterated model, quantized to MXFP4 (.weight packed 2-values-per-byte uint8 + .weight_scale uint8 E8M0)
• MTP (multi-token prediction) module weights, stored unquantized in original BF16 (the untouched weights from the base model, InternScience/Sage-A1.5) in model-mtp.safetensors
• model-mtp-mxfp4.safetensors: the same MTP module quantized to MXFP4, provided for runtimes whose MTP loader expects the packed quantized weights (see below)
• Vision tower weights kept in BF16 (excluded from quantization)
• Router gates, shared-expert sigmoid gate, layernorms, QK-norms, conv1d/SSM state params, embeddings and lm_head kept in BF16
• Full config set: config.json (with quantization_config for Quark), tokenizer files, processor/preprocessor configs, chat template, generation config

Serving with vLLM

vllm serve /path/to/this/repo

The quantization_config in config.json is picked up automatically (quant_method: "quark"). For multi-token prediction (MTP) speculative decoding:

vllm serve /path/to/this/repo \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

MTP loading per runtime

model-mtp.safetensors (canonical, in the index) holds the MTP module as unquantized BF16 — the original base-model weights, no quantization roundtrip.

• Radiance fork (magiccodingman/vllm-radiance) with RADIANCE_QUARK_BF16_MTP=1 (the fork default): loads the MTP as plain BF16 — works out of the box with this repo as-is.
• Standard vLLM with quant_method: quark: the Quark MTP loader expects the packed quantized weights. Swap in the provided quantized file before serving:

cd /path/to/this/repo
mv model-mtp.safetensors model-mtp.safetensors.bf16
cp model-mtp-mxfp4.safetensors model-mtp.safetensors
cp model.safetensors.index.mxfp4-mtp.json model.safetensors.index.json
vllm serve .

(model.safetensors.index.mxfp4-mtp.json is the index variant whose MTP .weight_scale entries are present, matching the packed file.) Verified on ROCm MI300X (vLLM 0.29.1 ROCm build): load + generation + MTP speculative decoding (mean acceptance length ~1.8).

Quantization details

• Weights: MXFP4 (E2M1), block size 32 along the input (K) dimension, E8M0 power-of-two group scales (OCP MX standard)
• Activations: dynamic per-32-group MXFP4 quantization (W4A4) at runtime
• MoE expert and shared-expert FFNs quantized per-expert (per-expert checkpoint names preserved)
• Quantized with aiter.ops.triton.quant.dynamic_mxfp4_quant via vllm.model_executor.layers.quantization.quark.utils.quark_quantize_weight_to_mxfp4 (AMD Quark 0.12, ROCm)