d1-Omni-2B-MLX-8bit
8-bit MLX version of d1-Omni-2B for Apple Silicon. The weights take 2.71 GB.
d1-Omni-2B is a decision model for text, pictures, video, and audio. It answers named, typed questions over a state with calibrated probabilities in one forward pass, without generating tokens. Questions can be written or spoken.
The language model uses 8-bit affine quantization with groups of 64. The vision tower, audio encoder, projectors and norms stay in FP16. Against the original checkpoint run in float32, it kept all 59 test answers and moved probabilities by at most 0.008.
Find architecture and usage details in the original model card.
Use with MLX-VLM
Requires MLX-VLM 0.7.11 or later:
pip install -U "mlx-vlm>=0.7.11"
d1-Omni-2B answers questions without generating text, so chat apps and mlx_vlm.generate cannot run it. Use model.system_one or the decide command below.
import numpy as np
from mlx_vlm import load
from mlx_vlm.utils import load_image
model, processor = load("LiquidAI/d1-Omni-2B-MLX-8bit")
# Text: several named questions over one state, answered in one pass
questions = {
"refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},
"team": {"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"billing": "Charges, refunds, invoices", "shipping": "Delivery and parcels",
"technical": "App or site faults"}},
"topics": {"type": "choice", "multiple": True, "instructions": "Which topics does the message raise?",
"criteria": {"billing": "Payments and charges", "delivery": "Shipping and parcels",
"login": "Signing in to the account"}},
"urgency": {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["low", "medium", "high"]},
}
out = model.system_one("I was charged twice and my parcel never arrived. Please refund one charge.", questions)
a = out["answers"]
print(round(a["refund"]["noul"], 2), a["team"]["choice"], a["topics"]["choices"],
a["urgency"]["legend"][str(int(a["urgency"]["score"]))])
# 0.94 billing ['billing', 'delivery'] medium
# Picture: the photo is the whole state
image = load_image("http://images.cocodataset.org/val2017/000000039769.jpg")
out = model.system_one(None, {
"cats": {"type": "number", "instructions": "How many cats are there?", "range": [0, 10], "step": 1},
"remote": {"type": "box", "instructions": "Where is the remote control on the left?"},
}, media=[image])
a = out["answers"]
print(a["cats"]["number"], {k: round(v, 2) for k, v in a["remote"]["box"].items()})
# 2.0 {'top': 0.15, 'bottom': 0.24, 'left': 0.06, 'right': 0.27}
# Audio: three short beeps over quiet noise, 16 kHz mono
rate = 16000
clip = np.random.default_rng(0).normal(0, 0.003, 4 * rate).astype(np.float32)
for start in (0.5, 1.5, 2.5):
clip[int(start * rate):int(start * rate) + 3200] += 0.3 * np.sin(2 * np.pi * 880 * np.arange(3200) / rate)
out = model.system_one(None, {
"beeps": {"type": "number", "instructions": "How many beeps are there?", "range": [0, 10], "step": 1},
"speech": {"type": "noul", "instructions": "Does someone speak in the recording?"},
}, media=[{"type": "audio", "audio": clip, "sampling_rate": rate}])
a = out["answers"]
print(a["beeps"]["number"], a["speech"]["noul"] > 0.5)
# 3.0 False
model.system_one returns {"answers": {name: answer}, "usage": {...}}, and model.system_one_batch takes several requests. The original model card describes every question type and answer field.
From the command line, the answers print as JSON:
python -m mlx_vlm decide --model LiquidAI/d1-Omni-2B-MLX-8bit \
--state "I was charged twice this month. Please refund one of the charges." \
--questions '{"refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"}}'
More options in the MLX guide.