ORCA — Llama-3.2-3B (Multinomial, seed 99, no rationale)
ORCA (Open-ended Response Correctness Assessment) scores the correctness of open-ended audio QA responses. Given a question, reference answer, candidate answer, and an LLM-generated rationale, it outputs a correctness score in [0, 1] and an uncertainty estimate.
Paper: ORCA: Open-ended Response Correctness Assessment for Audio Question Answering — accepted to TACL 2026
Code & usage: github.com/BUTSpeechFIT/ORCA
Training data: BUT-FIT/orca-audio-qa-annotations
Model details
| Property | Value |
|---|---|
| Base model | meta-llama/Llama-3.2-3B |
| LoRA rank / alpha | 128 / 128 |
| Loss function | Multinomial log-likelihood (5-class Likert) |
| Training seed | 99 |
| Training curriculum | Stage 1 (synthetic) → Stage 2 (LLM-judge) → Stage 3 (human) |
| Precision | bfloat16 |
| Rationale | NO |
Quick start
pip install git+https://github.com/BUTSpeechFIT/ORCA.git
hf download BUT-FIT/orca-llama-3.2-3b-multinomial-no_rationale --local-dir orca-llama-3.2-3b-no_rationale
orca-infer --model_path orca-llama-3.2-3b-no_rationale/model --data_jsonl your_data.jsonl --output_dir results/
See the repository for full usage, evaluation scripts, and the download_and_infer.py convenience script.
Citation
@article{sedlacek-etal-2026-orca,
title={ORCA: Open-ended Response Correctness Assessment for Audio Question Answering},
author={Sedl\'{a}\v{c}ek, \v{S}imon and Barahona, Sara and Bola\~{n}os, Cecilia and
Herrera-Alarc\'{o}n, Laura and Udupa, Sathvik and L\'{o}pez, Fernando and
Ferner, Allison and Lozano-Diez, Alicia and Yusuf, Bolaji and Kesiraju, Santosh and
Duraiswami, Ramani and \v{C}ernock\'{y}, Jan},
howpublished={Accepted to Transactions of the Association for Computational Linguistics},
year={2026},
url={https://arxiv.org/abs/2512.09066}
}
License
MIT License. See the repository LICENSE for details.