abacusai/Smaug-Decisions

---

State-of-the-art audio decisions in 4B parameters, and it sees and reads too.
60.1 Earshot audio skill, ahead of Gemini 4 Argon (59.4) and Gemini 3.8 Flash (50.5), and better calibrated than both.

1. Model Introduction

Smaug-Decisions is a 4B decision model that hears, sees and reads. A fine-tune of Google's Gemma 4 E4B-it by Abacus.AI, it answers a typed question about audio, an image or text with a calibrated probability per option, in one forward pass and with no generated text.

It is the only 4B decision model with results on all three input types. On audio it is ahead of the frontier APIs: 60.1 skill on Earshot, our 6,671-item audio decision benchmark, against 59.4 for Gemini 4 Argon and 50.5 for Gemini 3.8 Flash, and better calibrated than both (ECE 0.069 vs 0.123 and 0.204). On the public Jev Decision Index it scores 57.9 on images and 56.6 on text, the best Gemma 4 E4B on both.

import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"]))
from smaug_decide import SmaugDecide                                   # ships with this repo

model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions")
model("call.wav", "Has the speaker finished their turn?")               # -> {'Yes': 0.93, 'No': 0.07}

2. Highlights

• Ahead of the frontier APIs on audio, and better calibrated, so the probabilities work as thresholds.
• Hears, sees and reads: audio up to 30 s, images and text through one prompt and one readout.
• One pass, no decoding: about 200 ms per audio decision on an H100.
• Runs on phones: Smaug-Decisions-LiteRT is 3.45 GB, keeps 59.4 audio skill and answers a text question in about 0.6 s on an iPhone 14 Pro Max, fully offline.

3. Model Summary

4. Evaluation

Audio: Earshot

Earshot is our audio decision benchmark: 6,671 items and 17.2 hours of audio in four tracks: conversation dynamics (turn completion, next speaker, speaker count, speech activity; from VoxConverse, AMI and Smart Turn), external turn completion (Smart Turn v3.2 test), sound and music (genre, instrument, tempo, scene, sound events) and a small pilot track. Audio skill is the mean task skill per track, averaged over the tracks: 0 is guessing, 100 is perfect. No benchmark recording or item is in the training data.

Earshot table Bold = best in the row. Shading is relative within each row (darker = better). Every model scored on the same 6,671 items with the same harness; the APIs received the audio and returned JSON probabilities. Skill = (accuracy − chance) / (1 − chance), times 100.

Need it faster? Smaug-Audio-Decisions-0.7B scores 53.0 on Earshot at 9.7 ms; in front of this model as a cascade it matches this model's accuracy (60.1) at 130 ms instead of 210 ms.

Images and text: the public Jev Decision Index

The matrix at the top has the headline numbers: 57.9 on images and 56.6 on text, the best Gemma 4 E4B on both (Winnow-E4B, the other Gemma 4 E4B fine-tune on the boards: 48.1 and 41.7). Images: our run of the official vision kit (0.4, public sets) at 560 image tokens.

Full Decision Index breakdown Bold = best in the row. Other models: the public Decision Index board as of 2026-10-28 (text 0.4.1). Smaug-Decisions: our runs of the official Decision Index kit (text on a 3,000-row sample of the public suite, compared on the same 35 benchmarks); self-reported, not board submissions. — = no published result.

5. Training Approach

• Readout, not generation. The answer is read from the LM-head logits of the option letters at the answer position (tanh softcap 30, then softmax; select-all uses a sigmoid with one learned threshold). A decision costs one prefill.
• Decision-tuned and calibrated. Full fine-tune of Gemma 4 E4B-it on a broad mix of typed decision tasks over audio, images and text, with calibration as an explicit objective and options shuffled so the model learns the options, not their positions.

6. Limitations

• Decisions, not text: each call answers one typed question with a probability per option; the model does not write text.
• Per request: one audio clip of up to 30 s (longer audio keeps its last 30 s), one image at 560 image tokens, up to 20 options.
• Thresholds: set decision thresholds on your own data for high-stakes uses.

7. Usage

pip install "transformers>=5.20" torch soundfile pillow huggingface_hub

import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"]))
from smaug_decide import SmaugDecide

model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions")      # CUDA, Apple Silicon (MPS) or CPU, picked automatically
model("call.wav", "Has the speaker finished their turn?")                                  # predicate -> {'Yes': p, 'No': 1-p}
model("song.wav", "Which genre is playing?", ["rock", "jazz", "classical"])                 # choice -> probabilities sum to 1
model("room.wav", "Which sounds are present?", ["speech", "music", "alarm"], multi=True)    # select-all -> one probability each
model("receipt.jpg", "Does the total exceed $50?")                                          # images
model("The order never arrived.", "How upset is the customer?", ["calm", "annoyed", "angry"],
      descriptions=["neutral tone", "complains", "threatens to leave"], rules="Judge the tone, not the facts.")

from_pretrained downloads the checkpoint (about 16 GB). Any runtime can reproduce the readout: run the prompt in smaug_decide.render, take the LM-head logits of the tokens A..T at the last position, apply the softcap from readout.json, then softmax (one answer) or sigmoid(z - b) with b from readout.json (select-all). With plain Transformers, AutoModelForMultimodalLM.from_pretrained("abacusai/Smaug-Decisions", dtype="bfloat16") loads the weights.

On phones

Smaug-Decisions-LiteRT is this model in one 3.45 GB file for Google's on-device runtime LiteRT-LM, running on the phone GPU. At about a fifth of the size it keeps almost all of this model's quality. It runs in the Cascade app on iPhone and Android next to Smaug-Audio-Decisions-0.7B, which answers the easy audio questions first. App source: github.com/abacusai/Smaug-Decisions-Mobile.

Same items, same harness; the phone-build scores come from a PyTorch replica of the file's quantized weights. ^*10 of the 11 public vision sets (Winoground left out; this model's 11-set index is 57.9).

8. License

Apache 2.0, the same license as the base model, Gemma 4 E4B-it.

9. Citation

@misc{abacusai2026smaugdecisions,
  title  = {Smaug-Decisions: calibrated typed decisions over audio, images and text},
  author = {Abacus.AI},
  year   = {2026},
  url    = {https://huggingface.co/abacusai/Smaug-Decisions}
}

10. Contact Us

If you have any questions, please reach out at Abacus.AI.