mentat-sys1-v0.2
206/231 (89.18%) on JevBench Public-231. A one-pass 4B decision model built on Qwen3.5-4B, with native option probabilities and zero generated answer tokens.
Mentat v0.2 improves from 200/231 in v0.1 to 206/231, and answers
19 more questions correctly than the Qwen3.5-4B-based SemIF baseline.
JevBench Public-231
Higher is better.
| Model | Correct / 231 | Public accuracy |
|---|---|---|
| Plumb-4B | 207 | 89.61% |
| Mentat v0.2 (local) | 206 | 89.18% |
| Jev 1.13.0 | 200 | 86.58% |
| Mentat v0.1 (local) | 200 | 86.58% |
| Imajev-4B | 199 | 86.15% |
| JevK5 v0.2.0 | 197 | 85.28% |
| decider-4b v2 | 193 | 83.55% |
| Hopper | 190 | 82.25% |
| SemIF | 187 | 80.95% |
| Jobe Qwen3.5-4B | 187 | 80.95% |
External values are taken from the official JevBench v1.4.2.2 result artifact. Mentat was evaluated locally on the exact same 231 public files. These values show public accuracy, not the official composite JevBench Score.