qwen3.5-0.8b-slake-medvqa-lora

LoRA adapter for Qwen/Qwen3.5-0.8B fine-tuned for medical visual question answering on BoKelvin/SLAKE (English + Chinese). Research prototype; not validated for clinical use.

Model


Base model	`Qwen/Qwen3.5-0.8B` @ `2fc06364715b967f1860aea9cf38778875588b17` (`Qwen3_5ForConditionalGeneration`)
Architecture	ViT (12 blocks, patch 16, 2x2 merge) + 24-layer LM (18 Gated DeltaNet + 6 gated full-attention)
Adapter	LoRA r=8, alpha=16, dropout=0.05
Target modules	LM `self_attn.{q,k,v,o}_proj` (6 layers), `linear_attn.{in_proj_qkv,in_proj_z,out_proj}` (18 layers)
Trainable params	2,457,600 / 855,443,520 (0.2873%)
Frozen	vision tower, merger, MLPs, embeddings, lm_head

Data

BoKelvin/SLAKE @ a9083ce6c34ac3ffb17671a605962924d8a8f9e9, both languages, no translation. Split mode image_disjoint: the official test split is used unchanged; train/validation are rebuilt from official train+validation after removing every image that appears in test (official SLAKE splits share images), grouped by image hash, stratified by modality.

split	QA pairs	images
train	8542	391
validation	1549	71
test	2094	180

Excluded records: empty_answer=1, duplicate_record=14, image_shared_with_test=1828.

Training


Objective	answer-only causal LM loss (prompt and image tokens masked)
Batch	1 x grad. accum. 8 = 8
Optimizer	AdamW, lr 0.0001, weight decay 0.01, cosine schedule, warmup 0.03, grad clip 1.0
Epochs (max)	3 (early stopping patience 4 evals)
Precision	bf16 autocast, gradient checkpointing = True
Checkpoint selection	`eval_gen_exact_match` on validation
Image budget	65536-262144 px (<= 256 visual tokens)
Seed	42
Optimizer steps	1500
Best checkpoint	`checkpoint-1000` (eval_gen_exact_match = 0.78)
Train runtime	91.7 min
Hardware	NVIDIA GeForce RTX 5060 Ti (15.9 GB)
Software	torch 2.11.0+cu128, transformers 5.19.0, peft 0.18.1, CUDA 12.8

Evaluation

Official SLAKE test split (2094 QA, greedy decoding, max 32 new tokens). Accuracy = normalized exact match or numeric match; normalization = NFKC, case-fold, whitespace collapse, outer punctuation strip.

subset	n	accuracy	exact match	token F1	yes/no acc.
overall	2094	77.2	77.2	84.1	87.9
answer_type=CLOSED	836	83.0	83.0	88.3	87.9
answer_type=OPEN	1258	73.4	73.4	81.2	100.0
language=en	1061	78.8	78.8	82.2	89.0
language=zh	1033	75.6	75.6	86.0	86.8

Test answer-token loss: 0.456; empty-answer rate 0.0%; invalid-output rate 0.0%.

Image ablation (official test)

model / condition	accuracy	token F1
fine-tuned/correct	77.2	84.1
fine-tuned/none	30.8	40.9
fine-tuned/shuffled	39.6	49.4
base/correct	0.0	7.2
base/none	0.0	2.3
base/shuffled	0.1	5.0

Usage

import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16).cuda()
model = PeftModel.from_pretrained(base, "thealper2/qwen3.5-0.8b-slake-medvqa-lora").eval()
processor = AutoProcessor.from_pretrained("thealper2/qwen3.5-0.8b-slake-medvqa-lora")

question = "Does the picture contain liver?"
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Answer the following question about this image:\n" + question}]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = processor(text=[prompt], images=[Image.open("image.jpg").convert("RGB")], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Processor pixel budget used in training: size={'shortest_edge': 65536, 'longest_edge': 262144} (saved with the adapter).

Dataset attribution

SLAKE (Liu et al., ISBI 2021), CC BY 4.0: https://www.med-vqa.com/slake/

thealper2/qwen3.5-0.8b-slake-medvqa-lora