PaddleOCR-VL-1.6 — AWQ W4A16 (asymmetric)

INT4 weight-only quantization of PaddlePaddle/PaddleOCR-VL-1.6, made with llm-compressor 0.12.0 and stored in compressed-tensors pack-quantized format. vLLM loads it natively and picks MarlinLinearKernel.

Read the measurements below before deploying this. The quantization behaves well on most pages but has a failure mode the base model does not have, and it does not reduce the server's GPU footprint the way one might expect. Both effects were measured, not guessed.

What is quantized

Only the language model is quantized, and that is not a conservative choice — it is the only option vLLM accepts. vLLM builds this vision tower unquantized no matter what the checkpoint says, because of a divisibility gate in vllm/model_executor/models/siglip.py:

quantizable = (config.hidden_size % 64 == 0 and config.intermediate_size % 64 == 0)

PaddleOCR-VL's vision tower has intermediate_size = 4304, and 4304 / 64 = 67.25. A checkpoint with a quantized vision MLP therefore fails to load outright:

ValueError: There is no module or parameter named
'vision_model.encoder.layers.0.mlp.fc1.weight_packed' in SiglipVisionModel

The same 4304 also blocks a group_size = 128 scheme for mlp.fc2 on the transformers side. Quantizing only the vision attention (1152, divisible) does load and gets weights down to 1.27 GiB, but measured worse on OCR — see the table below.

Measured GPU memory

vLLM 0.29.0, one RTX 3090, --max-model-len 16384 --gpu-memory-utilization 0.40:

Same flags but deliberately small (--max-model-len 8192 --max-num-seqs 4 --gpu-memory-utilization 0.11 --enforce-eager):

The 0.34 GiB saved on weights went straight into the KV cache; the total footprint is the same. Quantization here buys more KV cache at equal footprint, not a smaller footprint. What shrinks the footprint is --gpu-memory-utilization; the weights only set the floor, and this checkpoint's floor is 0.34 GiB lower. That matters if you must fit under ~2.4 GB, and not otherwise.

This is the expected outcome for a 0.9B model: only 255M of its 906M parameters are quantizable, so INT4 gives ~1.2x on weights, not ~4x.

Measured OCR quality

Evaluated through the real PaddleOCRVL pipeline (paddleocr 3.7.0, PP-DocLayoutV2 layout detection on CPU, this checkpoint served by vLLM), not by prompting the model with whole pages — in production the pipeline sends one region crop per request with a task prompt. 20 held-out pages of getomni-ai/ocr-benchmark, scored against the reference page markdown after stripping markup from both sides (the pipeline emits HTML tables, the reference uses markdown pipe tables, so raw text similarity would measure the table dialect).

tokF1 is F1 over the token multiset. broken counts pages with a >=40-character run of one repeated character, or an output under half the BF16 length.

On the pages that come out intact, every quantized variant matches BF16 (this checkpoint: 0.807 vs 0.813 on its 18 good pages, −0.7%). The whole average gap is a handful of pages that fail outright: on a dense credit-card statement the model runs away into 000000... for thousands of characters, and on one IRS 1040 form it emits 336 characters where BF16 emits 12081, dropping the table body.

Six configurations were tried against this — page-level and crop-level calibration, symmetric and asymmetric, and holding out the layers usually blamed for 4-bit degeneracy. None removed it, and the two hold-out variants cost more memory than they saved. The failure looks intrinsic to running a 0.3B language model at 4 bits over long table generations; there is little redundancy to spare.

Recommendation: if you can afford BF16, use BF16 and control memory with --gpu-memory-utilization / --max-model-len, which is where the memory actually is. Reach for this checkpoint when you must fit under roughly 2.4 GB, and validate on your own documents first — especially dense financial tables. Consider adding a repetition penalty and a max_tokens cap.

Recipe

from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQMapping, AWQModifier

IGNORE = ["re:.*lm_head", r"re:model\.projector\..*", r"re:.*visual\..*"]

# PaddleOCRVLForConditionalGeneration is not in llm-compressor's AWQ mapping
# registry; the default mappings are written for a plain decoder stack.
MAPPINGS = [
    AWQMapping("re:.*language_model.*input_layernorm$",
               ["re:.*language_model.*self_attn.q_proj$",
                "re:.*language_model.*self_attn.k_proj$",
                "re:.*language_model.*self_attn.v_proj$"]),
    AWQMapping("re:.*language_model.*self_attn.v_proj$",
               ["re:.*language_model.*self_attn.o_proj$"]),
    AWQMapping("re:.*language_model.*post_attention_layernorm$",
               ["re:.*language_model.*mlp.gate_proj$",
                "re:.*language_model.*mlp.up_proj$"]),
    AWQMapping("re:.*language_model.*mlp.up_proj$",
               ["re:.*language_model.*mlp.down_proj$"]),
]

oneshot(
    model=model,                      # AutoModelForImageTextToText, transformers >= 5.2
    processor=processor,
    dataset=ds,
    recipe=AWQModifier(
        targets=["Linear"], scheme="W4A16_ASYM", ignore=IGNORE, mappings=MAPPINGS,
        # AWQ caches activations for every calibration sample at once, and the
        # vision tower sees patches before the 2x2 merge -- 24 GB is not enough.
        offload_device=torch.device("cpu"),
    ),
    data_collator=data_collator,
    num_calibration_samples=96,
)
model.save_pretrained(SAVE_DIR, save_compressed=True)   # without this the file does not shrink

Calibration: 96 document pages from getomni-ai/ocr-benchmark, each as a user turn (image + OCR:) with the ground-truth markdown as the assistant turn, capped at 512·28·28 pixels. recipe.yaml in this repo is what llm-compressor recorded.

One thing to know if you re-run this

llm-compressor writes quantization_config.ignore as absolute module paths of the runtime transformers tree (model.visual..., model.projector...). The checkpoint and vLLM name those same modules differently (visual..., mlp_AR..., language_model.lm_head), so the entries match nothing there, vLLM concludes the vision tower is quantized, builds weight_packed parameters for it, meets a plain weight in the file and dies with 'QKVParallelLinear' object has no attribute 'data'. The ignore list in this repo was rewritten as suffix regexes, which match in every namespace:

["re:.*lm_head", "re:.*visual\\..*", "re:.*vision_model\\..*",
 "re:.*mlp_AR\\..*", "re:.*projector\\..*"]

Usage

vllm serve Ar4ikov/PaddleOCR-VL-1.6 \
  --served-model-name PaddleOCR-VL-1.6-0.9B \
  --max-model-len 16384 --gpu-memory-utilization 0.4 \
  --limit-mm-per-prompt.image 1

from paddleocr import PaddleOCRVL

pipeline = PaddleOCRVL(
    vl_rec_backend="vllm-server",
    vl_rec_server_url="http://127.0.0.1:8000/v1",
)
for res in pipeline.predict("page.png"):
    print(res.markdown["markdown_texts"])

The pipeline asks for the model under the name PaddleOCR-VL-1.6-0.9B, hence --served-model-name. Prompts, if you drive the model directly: OCR:, Table Recognition:, Formula Recognition:, Chart Recognition:, Spotting:, Seal Recognition:.