DeepSeek-V4-Pro-Qwen3.5-9B — GPTQ-Pro W4 G64

Quantized using GPTQ-Pro — by groxaxo.

Local GPTQ-Pro quantization of Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B, pinned source revision 40c1fbe115db8a76ea0c594fc6759d39ffab4d15. No new fine-tuning is applied. The name and original model lineage follow the upstream publisher; see UPSTREAM_MODEL_CARD.md for its original claims, which this quantization does not independently verify.

Format and preservation

• Symmetric INT4, group size 64, desc_act=False, true sequential GPTQ-Pro quality_4bit() recipe.
• Group-aware reordering, activation-weighted MSE, strict FP32 Hessian staging, no numerical fallback, no GPTAQ/FOEM.
• All 32 decoder layers / 200 selected linear modules packed.
• 348 protected tensors, including all 15 MTP tensors and the vision tower, verified exactly equal to the BF16 source. These components remain at source precision.
• Weight files: 9.212 GB (decimal), versus 19.306 GB for the source.
• Calibration: 64 samples / 29,919 tokens from a mix of private text and general text. Calibration inputs are not distributed.

Serving speed: vLLM + Marlin

The recommended serving runtime is vLLM 0.24.0, selecting MarlinLinearKernel for AutoGPTQLinearMethod with quantization="gptq_marlin".

On one RTX 3090 under its existing 200 W power limit, BF16 and GPTQ-Pro weights were compared on the same card with 512-token prompts, fixed 65-token outputs, BF16 activations/KV, no prefix caching, warm-up excluded, and medians of three measured repeats:

Estimated decode throughput subtracts the median one-token call latency from the median 65-token call latency, then counts 64 decode steps. It includes warmed engine/scheduler overhead. Batch 4 rates are aggregate across four requests, not per-request speed. This gives 2.57x / 2.45x faster estimated decoding within the matched vLLM comparison. Cold startup, compilation, other workloads and different power caps can change performance. See PERFORMANCE.json for raw repeated timings and exact settings.

A tested offline benchmark configuration is:

from vllm import LLM, SamplingParams

llm = LLM(
    model="groxaxo/DeepSeek-V4-Pro-Qwen3.5-9B-GPTQPro-W4-G64",
    dtype="bfloat16", quantization="gptq_marlin",
    tensor_parallel_size=1, language_model_only=True,
    max_model_len=4096, max_num_seqs=4, max_num_batched_tokens=2048,
    gpu_memory_utilization=0.90, enable_prefix_caching=False,
)
outputs = llm.generate(["Explain binary search."],
                       SamplingParams(temperature=0, max_tokens=128))
print(outputs[0].outputs[0].text)

The text quality comparison below was also run through vLLM/Marlin. Vision and accelerated MTP remain outside these tests.

Quality: BF16 versus Marlin on the same runtime

The quality measurements below use vLLM 0.24.0 for both the original BF16 model and this W4 checkpoint, with Marlin selected for the quantized run. They are measurements of the recommended serving path.

Distribution fidelity

Full-vocabulary, teacher-forced KL(original BF16 || Marlin W4) in natural-log units. Raw logits were captured from the engine's actual compute_logits calls without modifying their values. Both runs used BF16 activations/KV. Log normalizers and vocabulary-chunk metric accumulation used FP64, with no top-k approximation. Each captured NLL was checked against vLLM's returned prompt log probabilities to validate normalization and causal alignment.

Scored 24,576 next-token positions from 64 nonoverlapping WikiText-2 test windows, 512 tokens each with the first 128 used only as conditioning. See KL_REPORT.json for the dataset revision, public window offsets and selection seed.

These are selected-window continuation scores, not standard full-corpus WikiText perplexity. KL measures distribution fidelity; these tests do not establish general coding/math accuracy, vision quality, long-context behavior or accelerated MTP quality.

Provenance and license

PROVENANCE.json, WEIGHT_CHECKSUMS.json, KL_REPORT.json and PERFORMANCE.json describe the exact artifact and tests. The source publisher declares Apache-2.0, retained here with upstream attribution.