Arxo Materia EHRSQL 0.1 answers a clinician's question about hospital records by writing one SQL query against the eICU database: lab results, prescriptions, diagnoses, costs, lengths of stay. This is Arxo-Materia-0.8B-EHRSQL-0.1, a LoRA adapter for Qwen/Qwen3.5-0.8B.
It is the second Arxo Materia release, after Arxo Materia 0.1 on MedCalc-Bench.
EHRSQL as MedHELM runs it: 1,959 questions from the validation and test splits, scored by MedHELM's own evaluator. The model writes the query on its own.
| Model | Execution accuracy | Validation (755 questions) | Test (1,204 questions) |
|---|---|---|---|
| Arxo Materia EHRSQL 0.1 (0.8B), this adapter | 1309 / 1959 (66.8%) | 521 (69.0%) | 788 (65.4%) |
| Qwen3.5-0.8B, no training | 65 / 1959 (3.3%) |
One training run. 99.6% of the answers execute without an error. The untrained row uses the same prompt and the same token limit; it was scored with the evaluator's comparison outside helm-run, with a limit of 60 seconds per query.
EHRSQL ships its own evaluation script, evaluate.py, which processes every query before running it; MedHELM runs the queries as written. With evaluate.py the same answers of this model score 83.0% on the same 1,959 questions. The leaderboard below uses MedHELM's evaluator.
EHRSQL column of the MedHELM leaderboard, release v6.0.0 of 9 November 2026, rows measured on the same 1,959 questions.
| System | Trained on the task | Execution accuracy | Source |
|---|---|---|---|
| Arxo Materia EHRSQL 0.1 (4B) | yes | 67.5% | arxohq/Arxo-Materia-4B-EHRSQL-0.1 |
| Arxo Materia EHRSQL 0.1 (0.8B) | yes | 66.8% | this model |
| Claude Opus 5.5 | no | 63.2% | MedHELM leaderboard |
| GPT-6.1 Sol | no | 42.8% | MedHELM leaderboard |
| Qwen3.8 Max | no | 42.7% | MedHELM leaderboard |
| Gemini 3.8 Flash | no | 41.0% | MedHELM leaderboard |
The Arxo Materia models were trained on the training split of EHRSQL. The leaderboard models are general models that answer from the prompt alone.
The adapter attaches to AutoModelForImageTextToText, not to AutoModelForCausalLM. The prompt below is MedHELM's prompt for EHRSQL; the model was trained with it.
import re
import urllib.request
import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer
base = "Qwen/Qwen3.5-0.8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "arxohq/Arxo-Materia-0.8B-EHRSQL-0.1")
# The database schema: the CREATE TABLE statements of eicu.sql from EHRSQL
SQL_URL = ("https://raw.githubusercontent.com/glee4810/EHRSQL/"
"9eb39b5e1fd0e4e2ec7bc31208a768dcdf873c50/dataset/ehrsql/eicu/eicu.sql")
tables, current = [], None
for line in urllib.request.urlopen(SQL_URL).read().decode().splitlines():
line = line.strip()
if line.startswith("CREATE TABLE"):
current = [line]
elif current is not None:
current.append(line)
if line.endswith(";"):
tables.append("\n".join(current))
current = None
schema = "\n\n".join(tables)
INSTRUCTIONS = (
"You are a highly skilled AI specializing in medical SQL queries with SQLite expertise. "
"Given a database schema and a medical question, generate a valid SQL query to retrieve the required information.\n\n"
"Requirements:\n"
"- Output only the SQL query without explanations\n"
"- Return exactly one SQL statement ending with ;\n"
"- Use current time `2105-12-31 23:59:00` if needed\n"
"- Return empty string if the question is unanswerable\n"
"- Do not add additional text or comments"
)
question = "How many patients were admitted to the hospital this year?"
prompt_text = (f"{INSTRUCTIONS}\n\nMedical Question + Schema: -- Database Schema:\n{schema}\n\n"
f"-- Question:\n{question}\n\n\nSQL Query:")
messages = [{"role": "user", "content": prompt_text}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
answer = tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(re.search(r"<sql>\s*(.*?)\s*</sql>", answer, re.S).group(1))
The model writes one comment line that restates the question and then the query inside <sql> tags. The reported score uses greedy decoding, thinking switched off and a limit of 1,024 new tokens.
Outputs for all 1,959 questions, per-question scores from MedHELM's evaluator and checksums: github.com/arxohq/arxo-materia-ehrsql. The 4B model: arxohq/Arxo-Materia-4B-EHRSQL-0.1.
Arxo is in early access: hi@arxo.io.
Licence: Apache-2.0, the licence of the base model.
Qwen/Qwen3.5-0.8B