D1A-E2B-it QAT · Core ML

What it does: decisions, chat, and photos, voice clips and video, in Core ML. It is the Core ML conversion of JohnP1/d1a-e2b-it-qat-mlx-q4 v0.7: the same weights, the same adapter, the same temperature. 4.6 GB.

A small decision model for Apple devices: give it a document and a few questions, and it picks the answers. Each question is yes/no, multiple choice or a score; D1A returns a calibrated probability for every option in one forward pass, with no generated text, so every answer is a valid option. With its adapter switched off the same model is Gemma 4 E2B-it QAT and chats.

Versions

version	date	what changed	load it with
v0.7 (main)	2026-10-30 (Core ML files 2026-11-10)	fine-tuned on English and Japanese dialogue acts and intents: better at "did they agree?", "is that just thanks, or a request?" and "a command or just talk?"	this repository
v0.6	2026-10-25	the D1A adapter retrained on Google's QAT int4 trunk	`--revision v0.6`

Only the adapter, the pointer head and the calibration temperature (1.55 → 1.91) differ between the two. What v0.7 is better at, and what it is not, is measured on the MLX build's card: dialogue acts +15.9 points [+12.2, +19.7] on 498 held-out questions; intent classification unchanged; every other D1A suite pooled −0.38 (not significant); calibration a little worse on some suites.

Updating a v0.6 folder in place: delete its .compiled/ folder, or use this version's reference runner, which compiles a package again whenever its content changed. A v0.6 runner keeps using the models it compiled before.

How the v0.7 files were made

The adapter is a separate fp16 branch in each trunk_* package, so v0.7 is the v0.6 packages with only those weights replaced:

every one of the adapter's 410 tensors was found exactly once in the v0.6 packages, byte for byte as the converter stored it (A, and B × alpha/r, in fp16); nothing else in the packages has their shape;
the v0.7 tensors were written in their place. Every changed byte lies inside those tensors, and every other file is byte-identical to v0.6;
writing the v0.6 tensors back gives files identical to the published v0.6 packages (sha256, all five).

head.safetensors and d1a_config.json are the MLX v0.7 build's. The encoders, lm_head, the embeddings, io.json and the goldens are v0.6's, unchanged: the adapter touches only the text trunk's projections. The reference runner changed in one place, the compiled-model check above.

Status: what it does not do yet

Verified on a Mac only (Apple M1 Max, macOS, the GPU). Not tested on an iPhone or iPad yet.
The Neural Engine is not used: every model is set up for the GPU (compute_units ALL or CPU_AND_GPU). The Neural Engine needs a different weight layout, not built.
No ready-made app or server: this repository holds the Core ML models, the host-side data and a Python reference runner (reference/d1a_coreml_qat.py). An app reproduces the host side; io.json lists every input and output and the host rules.
Limits: decisions take up to 1,536 tokens (two prefill sizes, 512 and 1,536); chat prompts up to 1,536 tokens and 2,048 positions in all (prompt plus answer); greedy decoding only in the reference runner.
Video is slow on the Mac: a 16-frame clip takes 0.9 s to encode and 8 to 11 s to answer (about 1,200 tokens at the 1,536 size) on the M1 Max. Photos and voice clips take about 0.4 s per request.
Photos, voice and video are answered less exactly than decisions: against the full-precision model, the largest change is 0.068 on the 28 media answers (mean 0.015, no top answer different) and 0.048 on decisions. Most of the media gap was the pad token's per-layer embedding row, which Gemma 4 reads at every media position: since tag v0.7-media-pad-row (jonpol01/d1a#394) media_pad_row.safetensors holds that row as the QAT checkpoint stores it, and the reference runner reads it instead of the 4-bit table's (before it: 0.117, one top answer different). An app's own port should do the same (io.json, host.media). Two long-video answers carry most of what is left (jonpol01/d1a#386).
Act on a "yes" only above a confidence threshold (v0.7): in dialogue-act tests it still reads some refusals and hedges as agreement; an app that acts on the answer should confirm when the top probability is below about 0.6.

Checks

On an Apple M1 Max, through this repository's own reference runner.

Re-run for v0.7 (against the MLX v0.7 build, on 2026-11-10):

Check	Result
The 28 decision requests of the v0.6 Core ML check (42 answers) against the MLX v0.7 build	the same answer on all 42; largest probability change 0.028, mean 0.005
The answers are v0.7's, not v0.6's	mean change 0.005 from MLX v0.7, 0.044 from Core ML v0.6; switching the adapter off moves an answer by up to 0.39
This Mac reproduces the v0.6 check	the v0.6 files give the v0.6 check's answers to within 0.00005
Photos, voice clips and video: the playground's 14 media requests (28 answers) against the MLX v0.7 build	missed the limit set before the check (largest change ≤ 0.05): largest change 0.051, on one voice clip's `intent` (same top answer, margin 0.33). 2 answers have a different top option, both near ties in the MLX build (margins 0.043 and 0.014); none of the 26 clearer answers changed. v0.6: largest 0.040, no answer different. Why: below
Chat on 2 photos (150 tokens each)	the same tokens as v0.6 (chat runs with the adapter off)
A v0.6 `.compiled/` folder left in place	compiled again; the answers are exactly v0.7's

Against the full-precision model (re-run for v0.7 on 2026-11-10: the PyTorch model in fp32 on a CUDA GPU, TF32 off, with Google's QAT trunk exact, the same adapter, and the QAT encoders fed fp32 inputs):

	decisions (the 28 requests, 42 answers): largest change, mean	photos, voice and video (14 requests, 28 answers): largest change, mean, top answers different
this build	0.048, 0.006	0.117, 0.029, 1
MLX v0.7	0.051, 0.008	0.140, 0.034, 2
PyTorch v0.7 (bf16)	0.050, 0.006	0.034, 0.006, 0

This build's encoders are not the cause: their output matches the full-precision encoders (per-token cosine ≥ 0.9996 on photos and video, ≥ 0.99999 on voice).
The gap is shared with the MLX build: given the same encoder output, the two builds' answers agree to within 0.021, and both move the same photo and video answers (where the parcel was left) the same way. One difference both share: they read the token and per-layer embeddings 4-bit, the reference at full precision. The cause is not found yet (jonpol01/d1a#386).
With 28 answers, one changed top answer against the MLX build's two is not a measured difference.

Carried over from v0.6 (the parts v0.7 did not change):

Check	Result (v0.6)
Decisions vs fp32 with this build's own host inputs (its 4-bit embedding rows): the trunk's own error	largest probability change 0.0017 (the MLX build 0.035)
Photo and video encoder against fp32 (18 images and videos)	per-token cosine ≥ 0.9996 (unchanged files; the same on the playground's 10 photos and videos against the full-precision encoders, 2026-11-10)
Voice encoder against fp32 (4 clips)	per-token cosine 1.0 (unchanged files; ≥ 0.99999 on the playground's 4 clips, 2026-11-10)
Chat: 8 photos, 427 answer tokens teacher-forced against fp32	final hidden state cosine ≥ 0.99977; 1 token of 427 different (a near-tie); chat runs with the adapter off, which v0.7 does not change
Chat: 1,000 tokens in one answer	stable (finite throughout)
The 4-bit text weights	all 276 quantized layers hold Google's QAT integers and scales exactly (no re-quantization; unchanged in v0.7)

The fp32 reference for chat is PyTorch on the CPU. On the same Mac, PyTorch's MPS backend gave a different fp32 answer on these image prompts, because of a PyTorch bug in MPS embedding lookups (jonpol01/d1a#353, pytorch/pytorch#188756), which zeroed the image start and end tokens' per-layer embeddings; d1a works around it since jonpol01/d1a#354.

What's inside

File	What	Size
`coreml/trunk_00-07` … `trunk_29-35.mlpackage`	the 35 decoder layers in 5 parts; each holds `prefill_512`, `prefill_1536` and `decode` (one token over a 2,048-position KV cache), sharing one copy of the weights	1.1 GB
`coreml/lm_head.mlpackage`	the tied output layer, int8 per row, 16 parts of 16,384 logits (chat)	0.40 GB
`coreml/vision.mlpackage`	Gemma 4's vision tower and the QAT projection: `image` (2,520 patches → 280 tokens) and `frame` (630 → 70, video)	0.34 GB
`coreml/audio.mlpackage`	Gemma 4's audio conformer and projection, fp32: 3,000 mel frames (30 s) → 750 tokens	1.18 GB
`embeddings.safetensors`	token and per-layer embeddings, 4-bit in groups of 64, read row by row on the host	1.55 GB
`head.safetensors`, `d1a_config.json`	D1A's pointer head, its calibration temperature (1.91) and the routing temperature (1.59) for requests that send `"use_case": "routing"` (jonpol01/d1a#385; tag `routing-t1.59`; not checked on this Core ML build)
`config.json`, `processor_config.json`, `tokenizer*.json`, `chat_template.jinja`	Gemma 4 E2B-it's configuration, processor and tokenizer
`io.json`	every model function's inputs and outputs, and the host rules
`reference/d1a_coreml_qat.py`	the reference runner (Python)
`goldens/`	host inputs for one decision request, one photo and one voice clip, for testing a port

How the weights are held:

The text layers: all 276 int4 linear layers exactly as Google's quantization-aware training left them (google/gemma-4-E2B-it-qat-w4a16-ct, revision 971342c0): the checkpoint's own integers and scales as Core ML block-wise int4.
The D1A adapter (LoRA rank 16) is a separate fp16 branch scaled by an input, lora: 1 for decisions, 0 for chat. At 0 the model is the QAT base exactly.
The encoders: the photo and video encoder runs in fp16, except the pixel path up to the patch embedding, which runs in fp32 (some image tokens are sensitive to fp16 pixel rounding). The voice encoder runs in fp32 for the same reason.

Run it

The reference runner uses the d1a package for D1A's request encoding and media decoding:

pip install "d1a[serve,media] @ git+https://github.com/jonpol01/d1a" "coremltools>=8"
huggingface-cli download JohnP1/d1a-e2b-it-qat-coreml --local-dir d1a-e2b-it-qat-coreml

v0.6 instead: add --revision v0.6.

import sys; sys.path.insert(0, "d1a-e2b-it-qat-coreml/reference")
from d1a_coreml_qat import Bundle
b = Bundle("d1a-e2b-it-qat-coreml")          # compiles the models once, into d1a-e2b-it-qat-coreml/.compiled
b.decide("Assistant: Shall I cancel the 3 pm meeting?\nUser: is that the one with the client?",
         {"answer": {"type": "choice", "instructions": "Did the user agree to the assistant's question?",
                     "criteria": {"yes": "Agrees: go ahead", "no": "Refuses, wants to wait, or corrects it",
                                  "unclear": "Neither: a counter-question, a hedge, or something unrelated"}}})
b.decide("", {"damaged": {"type": "noul", "instructions": "Is the parcel damaged?"}},
         media=("image", open("parcel.jpg", "rb").read()))       # or ("audio", wav bytes), ("video", mp4 bytes)
text, tokens = b.chat("What is this?", image=open("photo.png", "rb").read())

The first load compiles the Core ML models (about a minute on an M1 Max).

License and data

Apache-2.0. Base model: Gemma 4 by Google (Apache-2.0). Code: github.com/jonpol01/d1a, built on Kev by Jared Palmer (Apache-2.0). Intent data from MASSIVE 1.1 by Amazon (FitzGerald et al., 2022; CC BY 4.0), English and Japanese. Japanese decision data derived from JGLUE by Yahoo Japan Corporation and Waseda University (CC BY-SA 4.0). Pull-request data from NousResearch/hermes-agent (MIT); the labelled PR dataset itself, including the recent PRs, is private.

加载中…