qlm-map-classifier

A DeBERTa-v3 classifier that labels a K-12 mathematics student response with the misconception it exhibits, drawn from a 39-category grouped taxonomy derived from QLM's misconception ontology.

Read the Limitations section before using this model. Its recall is high and its precision is near chance: it flags most misconceptions present and is wrong about roughly half of what it flags. That trade-off determines where it can and cannot be used, and it is documented in full below.

Model details

Training data

On the example count. The figure is stated as approximately 8,200 rather than precisely, and that imprecision is deliberate. The number appears in the training results record as a free-text descriptor rather than as a count derived from a dataset manifest, and it cannot be reproduced from any current build script. A later rebuild of the training data on disk contains 10,530 examples under a different category grouping. Anyone needing an exact count for replication should treat this as unresolved.

EEDI's 4,370 responses are authentic student work. The remaining examples are derived from QLM's subject verticals and misconception graph rather than collected from students.

Evaluation

Component models are reported alongside the deployed combined model, at the same precision. The weakest figures are the most decision-relevant.

How to read these numbers

The deployed model over-flags. 86.2% recall against 49.7% precision means it catches most misconceptions that are present while being incorrect on approximately half of its positive predictions. It is a high-sensitivity, low-specificity instrument.

Accuracy of 49.5% is above chance but its baseline is undocumented. Uniform chance on a 39-class problem is 2.6%, so 49.5% is far from random. However, the class distribution of the training and test sets is not documented (see Limitations), so a majority-class baseline cannot be computed — and without it, 49.5% cannot be interpreted as skill rather than class imbalance. This is a real gap, not a rhetorical hedge.

The retrained component is functionally non-detecting. 0.5% recall on 308 examples across 23 categories means that component almost never fires. It is reported here because it contributes to the deployed combined model and its behaviour is a material limitation.

MAP-34 detection recall of 25.1% is the figure to weigh if the intended use is detection against the MAP taxonomy rather than the grouped-39 taxonomy. The model misses roughly three quarters of those cases.

Downstream integration result

Injecting this classifier's per-turn output into a generative tutor's prompt produced no measurable improvement in the tutor's correction match rate: 60% without labels versus 55% with, on 100 paired math word problems, judged by automated string matching. No significance test was computed; at n=100 a 5-point difference is within sampling error, and the result should not be cited as evidence of degradation. The design conclusion drawn internally was that a noisy per-turn signal is better integrated across a session into a state estimate than injected turn by turn — an architectural inference, not a result this experiment establishes.

Limitations

1. Precision is near chance. 49.7% on positive predictions. Roughly half of all flags are incorrect.
2. Mistake-location performance is poor. 16.3% on the deployed model, 12.9% on the retrained component.
3. MAP-34 detection recall is 25.1%. Approximately three quarters of MAP-taxonomy cases are missed.
4. One component has 0.5% recall and effectively never fires.
5. Class distribution is not documented. No majority-class baseline can be computed, so accuracy cannot be separated from class imbalance.
6. No inter-annotator agreement is documented. The label reliability of the non-EEDI training examples is unmeasured, so the ceiling on achievable performance is unknown.
7. Exact training-set size is not reproducible. See the note under Training data.
8. Per-class performance is not published. Only aggregate figures exist, so which of the 39 categories the model handles well is unknown.
9. No disaggregation by learner profile. Performance for English learners, students with learning differences, or students behind grade level is unmeasured. Differential performance across these groups is plausible and untested.
10. Mathematics and English only. No evaluation outside English-language K-12 mathematics.
11. Majority of training data is not authentic student work. 4,370 of ~8,200 examples come from EEDI; the remainder are derived from internal verticals and the misconception graph.

Out-of-scope uses

• Any determination about an individual student. Grading, placement, diagnosis, intervention assignment, or referral. With 49.7% precision, roughly half of such determinations would be wrong and none would be contestable.
• Any evaluation of a teacher.
• Unsupervised triage where a flag triggers action without human review. The over-flagging behaviour makes a human reviewer necessary, not optional.
• Per-turn injection into a generative tutor's prompt — see the integration result above.
• Detection against the MAP taxonomy, given 25.1% recall.
• Non-mathematics subjects, or non-English responses.

Intended uses

Research on automated misconception detection; a baseline for misconception benchmarks; aggregate, model-level analysis where the false-positive rate is tolerable; and a noisy observation feeding a session-level state estimator that integrates many signals rather than acting on any single one — the use for which it was built.

Citation

@misc{qlm-map-classifier-2026,
  title  = {qlm-map-classifier: a DeBERTa-v3 misconception classifier for K-12 mathematics},
  author = {Srivastava, Kumar Sumbhav},
  year   = {2026},
  note   = {Quantum Learning Machines, Inc.},
  url    = {https://huggingface.co/QuantumLearningMachines/qlm-map-classifier}
}