Qwen2.5-VL-7B-Medical-LoRA (QLoRA experiment — negative result)

Personal experiment (February 2026). I trained a QLoRA adapter for Qwen2.5-VL-7B-Instruct to extract clinical information from document images. On my small evaluation the fine-tuned model scored lower than the base model (73.7% vs 83.5%). I'm publishing it with the full numbers because the reasons it failed are the most useful part.

If you need this task, use the base model in NF4 without this adapter.

Code: sarathi-aiml/quantization-pipe

Results at a glance

Measured on one NVIDIA DGX Spark (GB10). Latency comes from the 5 benchmark images with torch.cuda.synchronize(). NF4 was faster than FP16 here. That result is specific to this hardware and software stack (GB10, PyTorch nightly), and I didn't check it on other GPUs. I never ran the accuracy eval on FP16, so I can't say how much accuracy NF4 itself costs.

What I'd flag about this experiment

1. The evaluation is very small. It uses 5 synthetic images that I rendered myself (110 expected terms, 125 expected values), scored with case-insensitive substring matching. A single image changes the average a lot. Treat these numbers as a sanity check, not a benchmark.
2. The fine-tune hurt structured documents. Three of the five cases scored exactly the same. The whole drop came from the two tabular cases: ICU flowsheet 100% → 72.4%, complex lab panel 82.9% → 61.5%.
3. Likely causes (hypotheses, not ablated):
   • Train/test mismatch: about 912 of the 1,000 training samples were narrative MTSamples transcriptions rendered as plain text images. The eval documents are dense tables and flowsheets.
   • Low-resolution training images: they were resized to 256 px max, so small table text is hard to read.
   • Truncated targets: responses were capped at 600 characters and sequences at 512 tokens, which teaches short answers.
   • No validation during training: I loaded 100 validation samples but disabled evaluation (eval_strategy="no"), so I couldn't see the regression until afterwards.
4. The LoRA also reached the vision tower. I froze the vision encoder before calling get_peft_model. But the target names (gate_proj, up_proj, down_proj) also match the MLPs in visual.blocks.*, so PEFT added trainable LoRA layers there. The adapter file shows 161,480,704 LoRA parameters in the language model and 28,876,800 in the vision blocks' MLPs. Next time I'd restrict targets with a regex to the language model.
5. The total parameter count printed during training was wrong. The log reported 4.88B total parameters and "3.9% trainable". bitsandbytes packs two 4-bit weights per byte, so numel() undercounts the quantized layers. Against the real ≈8.3B parameters, the 190.4M LoRA parameters are about 2.3%.
6. Merging didn't save memory. After merge_and_unload() on the 4-bit model, allocated memory went from 5.7 GB to 11.3 GB. I didn't dig into why. Keeping the adapter unmerged, or merging into a BF16 base and then re-quantizing, are the alternatives to test.
7. Some quantization paths failed in my environment. AutoAWQ failed because PytorchGELUTanh was removed in transformers 4.57. auto-gptq wouldn't build on this aarch64 / CUDA 12.8 setup. I used bitsandbytes NF4 instead.

What I'd do next: train on real tabular clinical forms at higher image resolution, without truncating targets, with LoRA limited to the language model, a per-epoch validation loss, and a larger held-out eval (hundreds of documents with field-level F1).

Per-case results (base NF4 vs NF4 + adapter)

Combined accuracy = (term accuracy + value accuracy) / 2. Matching is case-insensitive substring.

Where the fine-tuned model failed: in the ICU flowsheet it missed 15 numeric values, including FiO2, PEEP, MAP, CVP, pH, pCO2, and platelets. In the lab panel it missed ALT, AST, ALP, GGT, and LDH.

Memory and latency (all configurations)

Load time: FP16 88.7 s, NF4 92.7 s, NF4 + merge 92.7 s.

Training details

Loss: 1.754 at step 5, then 0.730 at step 120 (last logged step). The average training loss over the run was 0.996.

Environment

GB10 is compute capability 12.1 (sm_121). At training time that needed a PyTorch nightly (2.11.0.dev20260206+cu128) plus a small compatibility patch that routes integer prod / cumprod to the CPU. Other versions: transformers 4.57.6, PEFT 0.18.0, bitsandbytes 0.49.1, accelerate 1.12.0, datasets 4.4.2, Python 3.10.19, CUDA 12.8.

Usage

Recommended: base model in NF4 (no adapter)

import torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration, BitsAndBytesConfig
from qwen_vl_utils import process_vision_info
from PIL import Image

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct", quantization_config=bnb_config, device_map="auto",
).eval()
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

image = Image.open("medical_document.png")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Extract all medical diagnoses, medications with dosages, and vital signs from this document."},
    ],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs,
                   padding=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

To reproduce the fine-tuned numbers: add this adapter

from peft import PeftModel
model = PeftModel.from_pretrained(model, "sarathi-balakrishnan/Qwen2.5-VL-7B-Medical-LoRA")
model = model.merge_and_unload().eval()   # this is how the 73.7% / 11.3 GB numbers were measured

Example prompts used in testing:

• "Extract all lab values with their results, units, reference ranges, and flags from this laboratory report."
• "Extract all ventilator settings, hemodynamic parameters, drip rates, lab values, intake/output totals, and diagnoses from this ICU flowsheet."
• "Extract all prescribed medications with their dosages, directions, quantities, and refill counts from this prescription."

Reference API (GitHub repo)

The GitHub repo includes a small FastAPI server (/health, /extract, /analyze, /benchmark) and a Dockerfile. It passed my 5 smoke tests on the GB10. It is a demo, not a hardened service. Note: the server currently loads this adapter. Given the results above, switch it to the base NF4 model.

Intended use and limitations

• Research and education only. Not for clinical decision-making, diagnosis, or patient-facing use. Every output needs review by a qualified person.
• Evaluated only on 5 synthetic, cleanly rendered English documents. Not validated on real scans, handwriting, stamps, or noisy images.
• Can hallucinate or omit terms and values. The fine-tuned version does this more often than the base model on tabular documents.
• English only; US clinical abbreviations.

License

Apache-2.0, same as the base model. The training datasets carry their own licenses.

Citation

@misc{qwen25vl-medical-lora-2026,
  title  = {Qwen2.5-VL-7B-Medical-LoRA: a QLoRA experiment on medical document extraction (negative result)},
  author = {Sarathi Balakrishnan},
  year   = {2026},
  url    = {https://huggingface.co/sarathi-balakrishnan/Qwen2.5-VL-7B-Medical-LoRA}
}