🧩 TranNhiem-DeepCompression-Qwen3.6-35B-A3B

_{The full Collaborative Deep Compression pipeline (conceptual). This release uses pruning (whole routed experts) and low-bit quantization (NVFP4); clustering is not applied (see §3).}

Paper: Collaborative Deep Compression for Foundation Models (link coming soon) · Base model: Qwen3.6-35B-A3B · Sibling release: TranNhiem-DeepCompression-Gemma4-26B-A4B-it

67% smaller (67 GB → 22 GB), +35% batched throughput, +49% KV-cache capacity — keeps 99.8% of the original performance on average (≥ 98.9% on each of 8 benchmarks).

⚠️ Format: NVFP4 (4-bit weights and activations) on the routed experts. NVFP4 is the native 4-bit format of NVIDIA Blackwell GPUs: workstation RTX PRO 6000 Blackwell, DGX Spark (GB10) and GeForce RTX 50-series. Use vLLM ≥ 0.30.1. All numbers on this card were measured on GB200; other Blackwell GPUs should run the model, but their speed and kernel support were not tested (workstation/consumer Blackwell uses different FP4 kernels from B200/GB200). With older vLLM the experts fall back to a slower weight-only path (measured). Pre-Blackwell GPUs (Hopper, Ada, Ampere) have no FP4 tensor cores.

1. 📖 Model Introduction

This is Qwen3.6-35B-A3B (a mixture-of-experts vision-language model) compressed with Collaborative Deep Compression (CDC): the compression stages are designed to work together, in a fixed order, and each is judged on the modalities the model serves, not on text alone.

Two stages were applied:

Expert pruning (pillar A): each MoE layer keeps the 240 of its 256 routed experts that matter most for real text, image and video traffic. The shared expert is untouched.
4-bit experts (pillar C): the remaining routed experts are quantized to NVFP4 (4-bit weights and activations). Attention, the shared expert, the router and the vision tower stay in bf16.

2. 📋 Model Summary


Base model	Qwen3.6-35B-A3B (`Qwen3_5MoeForConditionalGeneration`), image + video + text input
Architecture	MoE, 40 decoder layers (hybrid linear + full attention), top-8 routing plus a shared expert, expert intermediate size 512
Experts per layer	240 routed (from 256) + 1 shared
Precision	routed experts NVFP4 W4A4 (NVIDIA ModelOpt); attention, shared expert, router, vision tower bf16
Size on disk	22 GB (bf16 base: 67 GB)
Weights in GPU memory	21.4 GiB (bf16 base: 65.5 GiB)
Serving	vLLM ≥ 0.30.1 (native NVFP4 MoE kernel); Blackwell GPU for W4A4 (measured on GB200)

3. 🔬 Deep Compression Method

Pillar	Stage	Applied here	What it did
A	Pruning-aware compression	✅	Kept 240 / 256 routed experts per layer
B	Clustering-aware compression	❌	Not applied (see below)
C	Quantization-aware compression	✅	Routed experts → NVFP4 W4A4
D	Serving	✅	vLLM native NVFP4 MoE path

Pillar A: multimodal expert pruning. Each routed expert is scored by how much it contributes when it is used: its router weight times the size of its output, averaged over the tokens routed to it. The scores come from 256 text conversations plus 256 image, multi-image and video conversations. The 16 lowest-scoring experts per layer are removed, with no retraining; the kept experts carry 95% of the routing weight. The same recipe was first developed on Gemma-4-26B, and it transfers: keeping 240 alone is within ±0.4 points of the original on 7 of 8 benchmarks.

Pillar C: experts-only NVFP4. The routed experts hold almost all of the parameters and tolerate 4-bit weights and activations: experts-only NVFP4 stays within ±0.7 points of bf16 on the vision benchmarks. Attention and the shared expert stay in bf16.

Why not pillar B (weight clustering) here. Clustered (codebook) weights need a lookup-table matrix kernel that vLLM does not ship, so a clustered model could not be served at speed.

4. 📊 Evaluation Results

Accuracy (%) on the same serving stack. This model's row is the original score plus the mean paired change over 3 pruning-calibration seeds.

This model keeps 99.8% of the original model's performance on average (at least 98.9% on every benchmark) at about one third of the weight memory.

	MMLU	AI2D	ChartQA	MMBench	MMMU	MMStar	OCRBench	MVBench (video)
Original bf16 model	82.15	88.12	88.52	91.43	66.67	67.40	90.90	74.54
This model	81.66	87.92	88.17	91.18	65.96	67.98	90.63	74.85
Performance retained	99.4%	99.8%	99.6%	99.7%	98.9%	100.9%	99.7%	100.4%

Retained = this model's score ÷ the original's score. Values above 100% are within evaluation noise: treat them as "no measurable loss", not as an improvement.

Detailed change vs. the original, on the same serving stack (percentage points; paired item-by-item; thinking disabled; mean ± s.d. over 3 pruning-calibration seeds):

MMLU	AI2D	ChartQA	MMBench	MMMU	MMStar	OCRBench	MVBench (video)
−0.49 ± 0.19	−0.20 ± 0.18	−0.35 ± 0.40	−0.25 ± 0.22	−0.71 ± 0.77	+0.58 ± 0.32	−0.27 ± 0.25	+0.31 ± 0.27

MMLU costs about half a point. Every vision benchmark is within noise.
Re-evaluating one checkpoint moves these benchmarks by about 1 point, so differences below that are not meaningful.

5. ⚡ Efficiency

One GB200 GPU, tensor-parallel 1, vLLM 0.30.1rc1, GPU memory utilisation 0.85. Throughput is the mean of 3 runs, interleaved with the bf16 base in one session (64 requests, 256 generated tokens each).

	bf16 base	This model	Change
Batched text requests (req/s)	8.97	12.12	+35%
Batched image requests (req/s, two image sets)	19.16 / 12.91	26.03 / 15.85	+36% / +23%
Batched video requests (req/s)	0.79	0.79	≈
Single-stream decode (tokens/s)	298	313	+5%
Energy efficiency (tokens per joule)	12.8	pending
Weights in GPU memory	65.5 GiB	21.4 GiB	−67%
KV-cache capacity at the same memory budget (32 K context)	3.69 M tokens	5.51 M tokens	+49%
Max concurrent requests at 32 K context	113	168	+49%
Max concurrent requests at 128 K context	32.7	49.4	+51%

Single-stream speed changes little: only about 3 B parameters are active per token, so the model is already fast. The gain is in batch throughput and in how many long-context requests fit in memory.
Video is limited by encoding the frames, which stays in bf16.

6. 🚀 Deployment

pip install "vllm>=0.30.1"
vllm serve trannhiem/TranNhiem-DeepCompression-Qwen3.6-35B-A3B --max-model-len 32768

Hardware: native 4-bit (W4A4) needs a Blackwell GPU (B200 / GB200 / B300, RTX PRO 6000 Blackwell, DGX Spark, RTX 50-series). Measured on GB200 only.
No MTP head: the pruned checkpoint does not include the base model's multi-token-prediction weights, so MTP speculative decoding is not available.

7. 💻 Usage

Qwen3.6 reasons by default. Our evaluations disable thinking; enable it if you want reasoning traces.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="trannhiem/TranNhiem-DeepCompression-Qwen3.6-35B-A3B",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/document.png"}},
        {"type": "text", "text": "Summarise this document in three bullet points."},
    ]}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    max_tokens=512,
)
print(resp.choices[0].message.content)

8. ⚠️ Limitations

All measurements are on one GPU type (GB200). Speed and energy on other hardware will differ.
Accuracy is reported on 8 benchmarks (knowledge, diagrams, charts, general and expert vision, OCR, video) with thinking disabled. The model was not re-evaluated for safety beyond the base model.
Pruning removes 16 routed experts per layer permanently. Rare skills that only those experts served may degrade more than these benchmarks show.

9. 📜 License

Apache-2.0, inherited from the base model (license). This is a modified version of Qwen3.6-35B-A3B: 16 routed experts per layer were removed and the remaining routed experts were quantized to NVFP4.

10. 📝 Citation

@article{cdc2026,
  title  = {Collaborative Deep Compression for Foundation Models},
  author = {Tran, Van Nhiem},
  year   = {2026},
  note   = {Paper link coming soon}
}

11. 📬 Contact

Trần Nhiệm · rick.tvn@foxconn.com WhatsApp: +886 934 311751