Qwen3.6-35B-A3B Abliterated NVFP4 (ModelOpt)
Abliterated and NVFP4-quantized version of Qwen/Qwen3.6-35B-A3B. Optimized for NVIDIA Blackwell (DGX Spark / GB10).
Performance
2x NVIDIA DGX Spark (GB10, SM121 Blackwell) — TP=2, CUDA Graphs
| Metric | NVFP4 TP=2 | FP8 TP=2 | Improvement |
|---|---|---|---|
| Decode tg128 c1 | 83.21 tok/s | 77.31 tok/s | +8% |
| Peak tg128 c1 | 84.00 tok/s | — | — |
| Decode tg128 c4 total | 194.54 tok/s | 90.58 tok/s | +115% |
| Peak tg128 c4 | 249.33 tok/s | — | — |
| Prefill pp2048 c1 | 8,033 tok/s | 5,636 tok/s | +43% |
| TTFT pp2048 c1 | 234 ms | 320 ms | -27% |
| Model size | 20 GB | 35 GB | -43% |
| KV cache available | 81.6 GB | 74.5 GB | +10% |
Single DGX Spark — CUDA Graphs
| Metric | Value |
|---|---|
| Decode tg128 c1 | 63.21 tok/s |
| Decode tg128 c4 total | 143.00 tok/s |
| Peak tg128 c4 | 178.67 tok/s |
| Prefill pp2048 | 7,083 tok/s |
What's in this model?
- Abliteration: Refusal directions removed from layers 14-26 using norm-preserving biprojected abliteration
- NVFP4 Quantization: Quantized with NVIDIA ModelOpt (v0.43.0) using 256 calibration samples from Open-Platypus, batch_size=64
- Format: ModelOpt FP4 — compatible with vLLM (
--quantization modelopt_fp4)
Model Details
- Base Model: Qwen/Qwen3.6-35B-A3B (Apache 2.0)
- Architecture: MoE, 35B total params, 3B active per token, 256 experts (8 routed + 1 shared)
- Quantization: NVFP4 on language model linear layers. Vision encoder, gates, conv1d, lm_head excluded.
- Size: ~20GB (vs 67GB BF16, vs 35GB FP8)
- Context: 32K (configurable)
Usage with vLLM
Single Node
vllm serve joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt \
--quantization modelopt_fp4 \
--trust-remote-code \
--dtype auto \
--kv-cache-dtype fp8 \
--attention-backend flashinfer \
-tp 1
Multi-Node (2x DGX Spark)
vllm serve joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt \
--quantization modelopt_fp4 \
--trust-remote-code \
--dtype auto \
--kv-cache-dtype fp8 \
--attention-backend flashinfer \
--enable-prefix-caching \
--enable-chunked-prefill \
-tp 2 --nnodes 2 --node-rank 0 \
--master-addr <HEAD_IP> --master-port 29501
Important Notes
- Vision: This checkpoint includes the
ConditionalGenerationconfig for compatibility, but vision weights are not included. The visual encoder dimensions (4304) are not NVFP4-aligned for TP>1. For vision support, use the base FP8 model. - First startup: FlashInfer JIT compilation takes ~25-30 minutes on first run. Subsequent starts use cached kernels (~5-7 minutes).
- Multi-node timeout: The
mpdistributed backend requires patchingstore_timeoutinvllm/distributed/utils.pyfrom 300s to 1800s for first-run JIT compilation across nodes. - FLASHINFER_NVCC_THREADS: Set to 16 for faster JIT compilation on multi-core systems.
Quantization Recipe
Quantized on NVIDIA H200 via Vast.ai in 7.4 minutes ($3.31):
import modelopt.torch.quantization as mtq
quant_config = mtq.NVFP4_DEFAULT_CFG
model = mtq.quantize(model, quant_config, forward_loop=calibrate_loop)
# calibrate_loop: 256 samples, batch_size=64, max_seq_length=1024
Abliteration Details
- Tool: jim-plus/llm-abliteration
- Measurement: Projected mode with flash attention on BF16 model
- Layers: 14-26 (strongest refusal signal, signal quality 0.14-0.22)
- Scale: 1.0, Sparsity: 0.0
Hardware Tested
- 2x NVIDIA DGX Spark (GB10 Blackwell, 128GB unified memory, 200Gb/s QSFP RoCE)
- NVIDIA H200 (144GB HBM3e) — validation only