A fast, single-pass decision model for medical multiple-choice questions. A frozen language model
(Qwen/Qwen3-8B) turns a question and each answer option into vectors, two small trained heads project them
into a shared 512-dimensional space, and the option most similar to the question is chosen. There is no
step-by-step reasoning and no text generation. Only the two heads and one scale value are trained (see below).
Status: research prototype. It answers about 46.0% of held-out four-option questions correctly, against 25% for random guessing. It is not suitable for clinical or any real-world medical decision.
Evaluated on 3842 questions from the labelled MedMCQA validation split (the official test split has no answers). Each question is scored only against its own options (chance is 25.0%). The statistical margin for one run is about +/-1.6 points.
| Method | Accuracy |
|---|---|
| Random guessing | 25.0% |
| Question-blind baseline (most often correct answer in training, ignores the question) | 28.1% |
| No-head baseline (cosine on the raw frozen vectors, no training) | 34.0% |
| This model, mean +/- std over 3 seeds | 46.0% +/- 0.2 |
| This model, published seed (44) | 46.2% |
| Slice (mean over seeds) | Accuracy |
|---|---|
| Average over the 21 subjects | 47.4% |
| Questions phrased with EXCEPT / NOT | 34.9% |
| All other questions | 48.7% |
Per-subject numbers are in results.json. Shuffling the questions against the options drops accuracy to chance,
so the model does use the question.
Qwen/Qwen3-8B, loaded in bfloat16 and never updated.This question: "{text}" is answered in one word: " and each option as This answer: "{text}" means in one word: "
({text} is replaced by the text).Linear - GELU - Linear - LayerNorm - GELU - Linear with hidden size 2048 and output size 512.import sys
from pathlib import Path
from huggingface_hub import hf_hub_download
sys.path.insert(0, str(Path(hf_hub_download("pavanmantha/clm-system1-medmcqa-qwen3-8b", "clm_inference.py")).parent))
from clm_inference import SystemOneModel
model = SystemOneModel.from_pretrained("pavanmantha/clm-system1-medmcqa-qwen3-8b")
options = ["Vitamin A", "Vitamin B12", "Vitamin C", "Vitamin D"]
print(options[model.choose("Scurvy is caused by deficiency of which vitamin?", options)])
It needs torch, transformers, safetensors and huggingface_hub, and downloads Qwen/Qwen3-8B on first use.
pavanmantha/medmcqa-clm, derived from
openlifescienceai/medmcqa (Apache-2.0): 134,416 train, 4,000 validation, 3,842 test questions. Options such as
"all of the above" were removed, and the test questions do not appear in train or validation.clm_heads.safetensors (trained heads), config.json (architecture, prompt and layers), clm_inference.py
(loading and prediction), results.json (all measured numbers).
Qwen/Qwen3-8B