RED-SNOW-5.3-FLASH-MLX-4bit

RED-SNOW-5.3-FLASH, Blackfrost-AI's offensive-security model, in a high-quality 4-bit MLX build for Apple Silicon

Tuned for long agentic contexts at max thinking, and built to run as fast as possible with TensorFold

48.2 tok/s warm decode at ~23k agentic context

53% lower KL to BF16 than a standard uniform 4-bit affine conversion

oQ4e mixed precision plus layer-by-layer DWQ distillation against the full BF16 model on workload-matched long-context data. Vision tower and DWQ-retrained MTP draft head included and working.

Lineage: zai-org/GLM-5.3-Flash-BF16 → Blackfrost-AI/RED-SNOW-5.3-FLASH-BF16 (offensive-security fine-tune) → this build (oQ4e mixed precision, then layer-wise DWQ against the BF16 weights).

* the uniform build decodes faster because it keeps fewer modules at higher bit-widths — it moves fewer bytes. This release trades ~9% decode speed for ~53% lower KL to BF16.

⚠️ Intended use and terms

RED-SNOW is built by Blackfrost-AI as an offensive-security model for authorized work — see the source card for its intended-use terms, which this quantization inherits.

It must not be used for anything illegal, in any jurisdiction, under any circumstances, including accessing systems or data without permission, deploying malware, stealing credentials or data, or disrupting services. The model reduces false refusals on security topics by design and its shipped persona is an offensive operator; it is not a safety control, an authorization service, a sandbox or a policy engine. Authorization, scope, credentials, network access and human approval must be enforced outside the model. Generated findings, code and commands may be wrong or fabricated: review them and run them only in isolated, authorized environments. You are solely responsible for your use of it.

---

Quality: workload-matched DWQ vs standard uniform 4-bit

Closeness to BF16 is measured by running BF16 (streamed a layer at a time) beside each build on identical tokens: KL = KL(BF16 ‖ build) per token, lower = closer; agreement = share of positions where the build's top token is BF16's.

Held-out general text (32 rows × 1,024 tokens; disjoint from DWQ training and iMatrix calibration):

~53% lower KL and ~8× smaller NLL gap to BF16 than uniform 4-bit.

Long agentic context (the gate set: 15 real coding-agent conversations, 3.3k–22.7k tokens, replies generated at production settings, scored on the model's own reply tokens vs BF16):

Held-out security eval set (18 defensive-security replies on local documentation contexts, disjoint from the DWQ security training rows): reply-token KL 0.01360, versus 0.01330 for the same build before DWQ — a paired change of +0.00030 with a 95% CI of +0.00006 to +0.00048, entirely above zero. The guard therefore flags this release as significantly worse on defensive-security reply fidelity, concentrated in the 8–16k-token band (+0.00045 there). This is the one measured regression in the build; it is disclosed rather than smoothed over, and it is why the strict gate verdict below is a fail.

Strict fidelity gate verdict: FAIL. The release rule for this conversion is "candidate KL must be lower than the pre-DWQ build on both held-out sets, with the security set as a regression guard." This build improves generic fidelity, ties on agentic, and regresses measurably on the security guard. It is released because the owner reviewed these numbers and chose to ship; treat the security-set delta as a real, if small, fidelity cost.

Performance

Measured on an M3 Ultra Mac Studio (256 GB) with TensorFold 0.6.5 (MLX 0.32.3), serving this checkpoint as-is.

Production agentic workload: a real coding-agent conversation (22,713 tokens with this release's default RED-SNOW template), 1,024-token replies, temperature 1.0 / top-p 0.95 / top-k 20, reasoning effort max, MTP drafting on. Cold = fresh server, nothing cached. Warm = same prompt again (prefix cached), mean of 8. Agent turn = the conversation plus one tool call and its tool output (median of 3).

Speculative decoding on the warm runs: ~2.0–2.3 tokens accepted per forward round (65–69% acceptance).

TensorFold's own short-reply bench (tools/bench_openai.py, thinking off, 64-token replies, median of 5), decode tok/s: code sampled 56.0 · chat sampled 46.5 · code greedy 71.8 · chat greedy 44.8.

Every drafted reply is token-for-token identical to serial decoding (9/9 cases exact on each of 2 runs).

Chat template and persona

Ships with RED-SNOW's own chat template (chat_template.jinja): it supplies the model's default operating prompt and turns thinking on by default. The template has no reasoning_effort variable — effort flags are ignored; thinking stays on through enable_thinking.

Quick start

TensorFold (text, MTP)

TensorFold version: run this build with TensorFold 0.6.5. TensorFold 1.0's native engine does not read mixed-precision builds like this one yet.

hf download grant-ai/RED-SNOW-5.3-FLASH-MLX-4bit --local-dir ~/models/RED-SNOW-5.3-FLASH-MLX-4bit
tensorfold serve ~/models/RED-SNOW-5.3-FLASH-MLX-4bit --context 40960

MTP drafting is on by default (--mtp-drafts 3 on Macs).

oMLX (text, vision, MTP)

omlx serve --model-dir ~/models --port 8000

with mtp_enabled: true and mtp_num_draft_tokens: 3 for this model in oMLX's model settings.

Both expose an OpenAI-compatible endpoint; see the model's generation_config.json for its default sampling.

Settings that matter

Technical details

How it was made

1. Source: Blackfrost-AI/RED-SNOW-5.3-FLASH-BF16 at revision c0a7b6e, every file size-checked against the Hub (203 files, 599 GiB).
2. oQ4e with the mlx-quantkit oQ path at group size 64, iMatrix collected on the real BF16 weights streamed a layer at a time, on 256 sequences × 1,024 tokens of the oQ code/chat/tool-use/reasoning/multilingual corpus. MTP head and vision tower preserved.
3. Layer-wise DWQ (distilled weight quantization): per decoder block, the BF16 block runs on the BF16 activations and the quantized block on the quantized model's own activations, and only the block's quantized scales and biases are trained to match. An update is kept only if it lowers held-out error and a next-block lookahead confirms it. A first DWQ trained on 1,024-token windows improved short text but lost on 3k–23k agentic replies, so the shipped build was retrained on whole long sequences (up to ~24k tokens): real TerminalBench agent trajectories, a defensive-security slice built from local documentation, generated deep-context rows, and general corpus rows — with replies written by the oQ4e build at production settings, a reply-token KL objective for the MoE block kinds, and a per-block-kind step-size probe. 39 blocks accepted training; the rest kept their oQ4e scales. Packed weights, norms, routers, embeddings and the vision tower are byte-identical to the oQ4e build: size, layout and speed do not change.
4. MTP draft head: three candidate heads were grafted onto this trunk and A/B-compared in the same session — the head trained on the first DWQ build won pooled tokens-per-round (+12.7% vs the shipped head, 2.25 vs 1.99) with drafted output token-for-token identical to serial decoding. The head only proposes drafts; TensorFold keeps a draft only if it equals the model's own token, so output is unchanged and only speed improves.
5. Verification: byte-level diffs at each stage; drafted-vs-serial exactness in TensorFold; the fidelity gate above (held-out generic + long agentic + security sets); production and canonical speed benches.

Limitations

• No standardized capability benchmark has been run on this build; quality here means closeness to the BF16 model and text likelihood, not task competence.
• The security-eval regression above is real: on this 18-row defensive set the build sits ~0.0003 KL/token further from BF16 than the pre-DWQ build it replaces, concentrated at 8–16k context. Whether that matters in practice depends on the workload — the magnitude is small, but it is statistically solid and disclosed.
• Refusal behaviour was not measured. The model is built to answer offensive-security requests in authorized contexts; quantization can shift that.
• The vision tower is carried through but no multimodal quality evaluation exists for the parent.
• Speed depends on machine, context length, and template.

Credits

• Z.ai for GLM-5.3-Flash.
• Blackfrost-AI for RED-SNOW-5.3-FLASH and its security corpus.
• oMLX for the oQ mixed-precision quantizer; TensorFold for the inference engine.

Disclaimer and licence

This conversion was made in collaboration with Blackfrost-AI, the authors of RED-SNOW-5.3-FLASH. It is not endorsed by, sponsored by or associated with Z.ai, oMLX or TensorFold, none of which reviewed or contributed to this repository or bear any responsibility for it.

Changes made to the upstream work: quantized to oQ4e mixed precision with an iMatrix from the BF16 weights; scales and biases refined by layer-wise DWQ against the BF16 weights on workload-matched long-context data; the MTP draft head replaced with a workload-trained head (winner of a same-body A/B against the original head and two distilled candidates); vision tower preserved. Upstream deployment kit, harness results and image assets are not included; see the parent repository.

Licence: MIT (LICENSE, inherited from the upstream model and weights). Your use is also subject to the terms of the upstream works.