Apex-35B-V1.0-W4A16-HGPTQ
4-bit Hadamard + GPTQ (W4A16, group size 64) quantization of Apexintelligence-AI/Apex-35B-V1.0 (5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68) across all 256 routed experts, shared experts, Attention/GDN layers, Vision, and the Native MTP speculative drafter. It compresses the 66.97 GiB BF16 MoE model down to 18.78 GiB (3.57× smaller) so it fits within a single 24 GB GPU.
- VRAM & GPU fit:
18.78 GiBtotal (20,160,901,544 B, saving48.19 GiBvs. BF16) across30,970text linears,775MTP linears, and110vision linears, withlm_headandmtp.fcin unrotated 8-bit (W8_g64). - Quantization recipe: 64×64 Walsh–Hadamard input rotation + MSE-initialized Cholesky GPTQ + 2-pass Hessian Coordinate Descent (
H-CD) + End-to-End FP32-masterRMSNorm/GDN KL tuning. - Fidelity vs. BF16:
0.0701weighted KL (0.1109mean KL,−3.80%PPL) onprobe-v4(math0.0436,instruction_following0.0507,science0.0517,chat0.0541),0.0625weighted KL (0.0575mean KL,−1.65%PPL) onprobe-guard,95.06%/88.30%isolated/E2E Native MTP Top-1 agreement (NLL 2.4931 → 2.4870), and97.35%/94.37%isolated/E2E vision Top-1 agreement onvision-probe-v2(99.26%/96.30%onvision-guard-v2).
Quality vs. BF16 Base Model (Apexintelligence-AI/Apex-35B-V1.0)
Teacher-forced evaluation comparing this checkpoint directly against the unquantized BF16 base model:
| Evaluation | BF16 Base (Apexintelligence-AI/Apex-35B-V1.0) |
Apex-35B-V1.0-W4A16-HGPTQ |
Notes |
|---|---|---|---|
| Safetensors Size | 66.97 GiB (71,903,645,408 B) |
18.78 GiB (20,160,901,544 B) |
3.57× compression (−48.19 GiB) |
Text Perplexity (probe-v4) (378 prompts, 840,187 tokens) ↓ |
5.1325 |
4.9373 (−3.80%) |
Chat, math, science, coding, tool use, multilingual |
Text Divergence vs. BF16 (probe-v4) (weighted KL / token mean KL) ↓ |
0.0000 |
0.0701 / 0.1109 |
Full-vocabulary FP32 KL (math 0.0436, IF 0.0507, science 0.0517) |
General & Multilingual Holdout (probe-guard) (76 prompts, 19,534 tokens) ↓ |
13.8270 |
13.5988 (−1.65%) |
Weighted KL 0.0625 (token mean KL 0.0575) |
| Native MTP (Speculative Drafter) (11,583 positions) | 100.0% Top-1 (NLL 2.4931) |
95.06% Isolated Top-1 · 88.30% E2E Top-1 |
Isolated MTP KL 0.0121, E2E KL 0.0758 (NLL 2.4870) |
Vision (vision-probe-v2) (96 images, 302 tokens) |
1.0000 cos sim · 100.0% Top-1 |
0.9881 cos sim · 97.35% Iso / 94.37% E2E Top-1 |
Isolated ViT KL 0.0169, E2E KL 0.0528 |
Vision Guard (vision-guard-v2) (48 images, 135 tokens) |
1.0000 cos sim · 100.0% Top-1 |
0.9912 cos sim · 99.26% Iso / 96.30% E2E Top-1 |
Isolated ViT KL 0.0035, E2E KL 0.0221 |
What Is Quantized
- Quantized (
31,857linear layers total):- Language (
30,970W4A16 linears +1W8_g64linear):130attention projections (self_attn,linear_attn) +120shared expert projections (mlp.shared_expert.{gate_proj,up_proj,down_proj}) +30,720routed MoE expert projections (40layers ×256experts ×3projections) with online block-Hadamard rotation (head_dim=64) + Cholesky GPTQ (group_size=64, MSE scale search, 2-pass H-CD) + End-to-End FP32-master Norm/GDN KL QAT, pluslm_headin unrotatedW8_g64. - Native MTP (
775W4A16 linears +1W8_g64linear):4self_attnprojections +3mlp.shared_expertprojections +768routed MoE expert projections (256experts ×3projections) (head_dim=64,group_size=64), plusmtp.fcin unrotatedW8_g64. - Vision (
110linears): All 27 ViT blocks (visual.blocks.*.{attn.qkv,attn.proj,mlp.linear_fc1,mlp.linear_fc2}) plus the 2 spatial merger projections (visual.merger.{linear_fc1,linear_fc2}).visual.blocks.*.mlp.linear_fc2(K=4304) uses per-channel Cholesky GPTQ +head_dim=16online Hadamard rotation (v_vision_fc2); all other Vision linears usegroup_size=64+head_dim=64.
- Language (
- Kept in BF16:
embed_tokens,linear_attn.in_proj_a,linear_attn.in_proj_b,mlp.gate,mlp.shared_expert_gate,visual.patch_embed.proj, norms, and biases.
- Static Hot-Expert Cache Rankings:
static_hot_cache_rankings.json(andruntime/static_hot_cache_rankings.json+runtime/expert_routing_profile.json) provides pre-profiled expert rankings (0..255ordered hottest-first per layer"0".."39").
- Default sampling (
generation_config.json):temperature 1.0, top_p 0.95, top_k 20(eos_token_id: [248046, 248044],pad_token_id: 248044).
Runtime & vLLM Notes
- Zero-patch stock
vLLM+ Hugging Facecompressed-tensorscompatibility:- Ships
vllm_warp_quant.py(auto_map) withvLLM 0.34+tp>=1SharedWeightParameterand Visionlinear_fc2boundary-straddling Hadamard support. - Supports native
qwen3_next_mtp/mtpspeculative decoding invLLM(--speculative-config '{"method":"mtp","num_speculative_tokens":1}').
- Ships
- Online Input Hadamard Rotation Across Attention, Shared Experts, and Routed MoE Experts:
- Every quantized linear in Text (
30,970linears) and Native MTP (775linears) — includingself_attn,linear_attn,mlp.shared_expert.{gate_proj,up_proj,down_proj}, andmlp.experts.0..255.{gate_proj,up_proj,down_proj}— has its weights pre-rotated offline along the input dimension ($W' = W H_{64} / 8$) prior to Cholesky GPTQ quantization (group_size=64,actorder="static"). - At runtime, apply block-64 Walsh–Hadamard transform ($x \mapsto x H_{64} / 8$) to the input activations before
gate_proj/up_proj(K=2048) and to the post-SwiGLU intermediate activations beforedown_proj(K=512), matchingquantization_config.transform_config.config_groups.v. - Only
mlp.gateandmlp.shared_expert_gateremain in BF16 (listed inquantization_config.ignore).
- Every quantized linear in Text (