Agnes-3.0-Flash DWQ 4-bit (MTP, No Vision)
Agnes-3.0-Flash is a 33B hybrid architecture (54 delta-recurrent + 18 global attention layers). This variant uses DWQ (Data-aware Weight Quantization) at 4-bit with a group size of 64, plus a native MTP (Multi-Token Prediction) head for speculative decoding. The vision tower has been stripped for text-only inference.
⚠️ Fixed 2026-10-16 — MTP head repair. The initial upload shipped a malformed MTP head that made the model fail to load in oMLX (
Expected shape (19456, 640) but received shape (21504, 640)). The broken head has been replaced with the correct one and the model now serves with working speculative decoding at ~30 tok/s (~2.7× over baseline). If you downloaded this model before that date, re-download it. Full details in MTP head repair.
What is DWQ?
DWQ is a data-aware quantization method that goes beyond simple round-to-nearest (RTN). Instead of blindly snapping each weight to the closest quantization grid point, DWQ considers the actual distribution of activations in the model. It uses this information to make smarter choices about how weights are scaled and shifted before quantization.
How DWQ compares to RTN
| Aspect | RTN | DWQ |
|---|---|---|
| Method | Round each weight to nearest grid point | Data-aware scaling + rounding |
| Quality at 4-bit | Noticeable degradation | Closer to bf16 baseline |
| Outlier handling | Outliers dominate the grid | Outliers are managed via data-aware scaling |
| Implementation | Trivial | Requires calibration data |
| Speed impact | None | None (same inference speed) |
The practical result: DWQ at 4-bit typically preserves more of the model's original capability than RTN at the same bit width. You get a smaller model without giving up as much quality.
Model Details
- Architecture: Qwen3_5ForConditionalGeneration (hybrid: GatedDeltaNet + full attention)
- Parameters: ~33B
- Quantization: DWQ 4-bit, group_size=64, affine mode (uniform — no per-tensor overrides)
- MTP head: 1 layer, FFN 19456 (enables speculative decoding)
- Vision: Stripped (text-only)
- Context length: 262,144 tokens
- Model size: ~18.5GB (quantized, no vision)
MTP head repair
The MTP head in the initial upload was malformed and the model would not load in oMLX:
mtp.gate_proj/up_projwere[21504, 640]— the main model's FFN is 19456mtp.down_projwas[5120, 4480]with scales[5120, 2352]— 2048 extra columns, giving an invalid group size
oMLX rejected it with Expected shape (19456, 640) but received shape (21504, 640) and [quantize] The requested group size 8 is not supported.
Fix: the MTP shard (model-00005-of-00005.safetensors) was replaced with the correct head from the working oQ4e graft — a uniform 4-bit, group-64 MTP head with FFN 19456 that matches the main model. Tensor names are identical, so the index weight map is unchanged.
Benchmarks
Measured on M3 Max 64GB with oMLX, 512-token generations, MTP depth 4:
| Metric | Value |
|---|---|
| Baseline (no MTP) | ~11.1 tok/s |
| With MTP (depth 4) | ~30 tok/s |
| MTP acceptance rate | 68–71% |
| Tokens per cycle | 2.4–2.7 |
| Speedup | ~2.7× |
| MTP overhead | ~1.5% of step time |
Depth-4 acceptance breakdown: d1 ≈ 74%, d2 ≈ 68%, d3 ≈ 67%, d4 ≈ 72%.
Note: ANE prefill is best disabled for this model on a 64GB machine — with it enabled the load footprint is ~31GB and the process can be OOM-killed; with it off the model loads at ~17.3GB.
Usage
With MLX-LM
from mlx_lm import load, generate
model, tokenizer = load("hermitdave/Agnes-3.0-Flash-DWQ-4bit-MTP")
response = generate(model, tokenizer, prompt="Hello, how are you?")
With oMLX
The model is served via oMLX with MTP speculative decoding enabled. Send requests to the /v1/chat/completions endpoint.
Attribution
- Base model: Agnes-3.0-Flash by Agnes AI
- Quantization: DWQ 4-bit (this repo)
- MTP head: grafted from the oQ4e build (Agnes native head, FFN padded to 19456)
- MTP repair & validation: Hermit Dave with Hermes Agent
License
Apache 2.0