AutoAWQ-format 4-bit quantized version of Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2.
This checkpoint keeps the same hybrid Qwen3.5 DeltaNet + softmax architecture and Qwen3.5 MTP head as the BF16 source, but exports an AutoAWQ-compatible W4A16 checkpoint for broader AWQ tooling and runtime compatibility.
The published folder includes:
model-00001-of-00011.safetensors ... model-00011-of-00011.safetensorsmodel.safetensors.index.jsonmodel.mtp.safetensorsquantization_config.jsonprocessor_config.jsonpreprocessor_config.jsonvideo_preprocessor_config.jsonLocal export was completed on 2026-03-31 on a single NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) with:
auto-round==0.10.2transformers==5.3.0vllm==0.17.1What was verified in that run:
quantization_config.json was written with quant_method=awqbits=4, group_size=128, sym=false, zero_point=truemodel.mtp.safetensors was restored into the output folderFull local vLLM serve validation for this exact AWQ v2 export is still pending.
AutoRound AutoAWQ export using W4A16 asymmetric group-wise quantization:
| Precision | Layers |
|---|---|
| INT4 weights + BF16 activations | most quantized linear layers |
| BF16 | lm_head, embed_tokens, self_attn.o_proj, DeltaNet linear_attn.out_proj, DeltaNet in_proj_a/in_proj_b, visual encoder, MTP sidecar |
AWQ details:
128sym=falsetruegemmArchitecture match with the BF16 source:
model_type=qwen3_564 text layersfull_attention_interval=4mtp_num_hidden_layers=1max_position_embeddings=262144No local benchmark slice is included yet for this AWQ v2 export.
The export completed successfully and the checkpoint layout is ready for upload, but full serve/runtime validation is still pending.
pip install -U vllm==0.17.1 transformers==5.3.0
Expected serving command:
vllm serve mconcat/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-AWQ-4bit \
--max-model-len 262144 \
--gpu-memory-utilization 0.85 \
--max-num-seqs 1 \
--skip-mm-profiling \
--reasoning-parser qwen3
With MTP enabled:
vllm serve mconcat/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-AWQ-4bit \
--max-model-len 262144 \
--gpu-memory-utilization 0.85 \
--max-num-seqs 1 \
--skip-mm-profiling \
--reasoning-parser qwen3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
This export is not intended for plain transformers inference.
Use a runtime that understands AutoAWQ-format checkpoints, such as vLLM with AWQ support.
| Framework | Supported | Notes |
|---|---|---|
| vLLM >= 0.17.0 | Expected | Intended serving path for this AutoAWQ export; exact local serve validation still pending |
| transformers >= 5.3.0 | No | Plain transformers is not the intended inference path for this AutoAWQ checkpoint |
| AutoAWQ-compatible runtimes | Expected | Export format is AutoAWQ-style quant_method=awq, version=gemm |
| SGLang | Unknown | Not verified for this export |
self_attn.o_proj and DeltaNet linear_attn.out_proj in BF16 rather than 4-bit.model-00009-of-00011.safetensors 2.5 GBmodel-00011-of-00011.safetensors 2.5 GBmodel-00002-of-00011.safetensors 2.1 GBmodel-00003-of-00011.safetensors 2.1 GBmodel-00006-of-00011.safetensors 2.1 GBmodel-00007-of-00011.safetensors 2.1 GBmodel-00001-of-00011.safetensors 2.1 GBmodel-00004-of-00011.safetensors 2.1 GBmodel-00005-of-00011.safetensors 2.1 GBmodel-00008-of-00011.safetensors 1.8 GBmodel-00010-of-00011.safetensors 872 MBmodel.mtp.safetensors 849 MB有文件?提交种子。
提交种子