d1-Omni-2B-GGUF
GGUF version of d1-Omni-2B for local inference on CPUs and GPUs.
d1-Omni-2B is a decision model for text, pictures, and audio, built on LFM3-Omni-2B. It returns calibrated answers in one forward pass, without generating tokens.
Find architecture and usage details in the original model card.
Use with llama.cpp
llama.cpp answers noul, choice and score questions over a state (text or JSON), pictures, a video, audio clips, and questions written or spoken. Videos are read as their frames, and video input needs ffmpeg in PATH. Multiple choice ("multiple": true), extraction, estimates and boxes are not supported yet.
llama-server -hf LiquidAI/d1-Omni-2B-GGUF:Q8_0 -b 8192 -ub 8192
A question is read in one batch, so -b and -ub must hold its state and media. A picture takes at most half of -ub: 8192 keeps the full budget of 4096 tokens per picture, smaller values lower it.
Then send questions to the /v1/systemone endpoint.
Text
curl http://127.0.0.1:8080/v1/systemone -H "Content-Type: application/json" -d '{
"state": "I was charged twice this month, please refund one of them.",
"questions": {
"refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},
"team": {"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"billing": "Charges, refunds, invoices", "technical": "App or site faults",
"fraud": "Suspected unauthorised use"}},
"urgency": {"type": "score", "instructions": "How urgent is this?",
"criteria": ["Can wait", "Today", "Blocking the customer now"]}
}
}'
Image + Text
curl -sL -o cats.jpg http://images.cocodataset.org/val2017/000000039769.jpg # two cats on a sofa
curl http://127.0.0.1:8080/v1/systemone -H "Content-Type: application/json" -d @- <<JSON
{
"state": "Photo attached to a pet-sitting request.",
"images": ["data:image/jpeg;base64,$(base64 < cats.jpg | tr -d '\n')"],
"questions": {
"pet": {"type": "choice", "instructions": "Which animals are in the photo?",
"criteria": {"cats": "Cats", "dogs": "Dogs", "birds": "Birds"}},
"sofa": {"type": "noul", "instructions": "Are the animals on a sofa?"}
}
}
JSON
Audio + Text
curl -sL -o jfk.wav https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav
curl http://127.0.0.1:8080/v1/systemone -H "Content-Type: application/json" -d @- <<JSON
{
"state": "Recording from a public event.",
"files": ["data:audio/wav;base64,$(base64 < jfk.wav | tr -d '\n')"],
"questions": {
"kind": {"type": "choice", "instructions": "What kind of utterance is this?",
"criteria": {"request": "A request to do something",
"question": "A question asking for information",
"speech": "A speech to an audience"}},
"calm": {"type": "score", "instructions": "How calm is the speaker?",
"criteria": ["Agitated", "Neutral", "Calm"]}
}
}
JSON
Video + Text
curl -sL -o reading.mp4 https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/4cba700bd771f44d72b549253da025c32e944d42/sample_demo_1.mp4 # a child reading a book
curl http://127.0.0.1:8080/v1/systemone -H "Content-Type: application/json" -d @- <<JSON
{
"state": null,
"files": ["data:video/mp4;base64,$(base64 < reading.mp4 | tr -d '\n')"],
"questions": {
"activity": {"type": "choice", "instructions": "What is the child doing?",
"criteria": {"reading": "Reading a book", "eating": "Eating", "sleeping": "Sleeping"}},
"glasses": {"type": "noul", "instructions": "Is the child wearing glasses?"}
}
}
JSON
Spoken question
A question can be spoken: its instructions are an input_audio part.
curl http://127.0.0.1:8080/v1/systemone -H "Content-Type: application/json" -d @- <<JSON
{
"state": "I was charged twice this month, please refund one of them.",
"questions": {
"refund": {"type": "noul",
"instructions": {"type": "input_audio", "input_audio": {"data": "$(base64 < question.wav | tr -d '\n')", "format": "wav"}}}
}
}
JSON
The state can be null when the pictures, the video or the audio are the whole state.