PitVQA Motion Model

Specialized model for temporal motion detection between surgical video frames.

Description

This model analyzes pairs of frames to detect instrument motion. It outputs structured motion annotations:

<motion instrument="suction" direction="left" magnitude="moderate"/>

Training

• Base: Qwen/Qwen2-VL-2B-Instruct
• Method: SFT with LoRA
• Dataset: Motion annotations from PitVis-2023

Usage

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel

base = Qwen2VLForConditionalGeneration.from_pretrained("Qwen/Qwen2-VL-2B-Instruct", ...)
model = PeftModel.from_pretrained(base, "mmrech/pitvqa-qwen2vl-motion")

# Provide two frames for motion analysis
messages = [{"role": "user", "content": [
    {"type": "image", "image": frame1},
    {"type": "image", "image": frame2},
    {"type": "text", "text": "Describe the instrument motion between these frames."}
]}]

Related Models

• pitvqa-qwen2vl-unified-v2 - Multi-task including motion (Stage 3)