iota-decide-4b GGUF

GGUF builds of macrocosm-os/iota-decide-4b, a 4B decision model by Macrocosmos, for llama.cpp. Run it on a laptop, on CPU or on a single GPU, behind a SystemOne-compatible /v1/systemone endpoint.

For reference, the original bf16 weights score 182/231. Scores are our own run on the JevBench v1.5 public split (231 questions). The official result is pending.

Run

Requires llama.cpp 0.7.0 or newer.

llama-server -hf macrocosm-os/iota-decide-4b-GGUF:Q8_0 -b 8192 -ub 8192 --port 9931

A decision model reads the whole input in one pass, so set the batch sizes (-b, -ub) at least as large as your longest request.

curl http://127.0.0.1:9931/v1/systemone -H "Content-Type: application/json" -d '{
  "state": {"ticket": "Customer asks for a refund 45 days after purchase.",
            "policy": "Refunds are allowed within 30 days of purchase."},
  "questions": {
    "eligible": {"type": "noul", "instructions": "Is the customer eligible for a refund under the policy?"},
    "route": {"type": "choice", "instructions": "Which team should handle this?",
              "criteria": {"refunds": "Refunds and billing team", "tech": "Technical support", "sales": "Sales"}},
    "urgency": {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["Low", "Medium", "High"]}
  }
}'

In this API, score criteria are a list of levels, lowest first.

Licence

Apache-2.0, the same as the original model. See its NOTICE for attributions.