Gemma-4-12B-it AEON Abliterated — MLX FP4 (mixed mxfp4 / mxfp8, compact)

The compact, fast, 16 GB-friendly Apple-Silicon build of AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-BF16. True 4-bit mxfp4 on the bulk of the decoder, 8-bit mxfp8 on the two residual-writers that carry the abliteration edit, and bf16 on the soft-capped head and the vision/audio projectors. Built and validated on a MacBook Pro M4 Pro (48 GB).

Target hardware: Apple Silicon (M-series), runs on 16 GB+ unified memory (peaks ~10.1 GB). Full multimodal (text + image + audio) via mlx-vlm.

Want maximum fidelity? See the near-lossless 8-bit sibling: …-MLX-8bit.

This is the smallest member of the MLX quant grid (9.3 GB) and the fastest single-stream — a high-quality compact build for Apple Silicon on a tight unified-memory budget. It stays fully coherent, keeps the abliteration intact (0/8 refusals on the harmful probe), and preserves the full vision/audio path. For the tightest possible match to BF16, the near-lossless MLX-8bit sibling is one click away.

⚡ Quickstart (Apple Silicon)

0 → running on a fresh Mac (no Python, no tools needed) — uv installs a correct Python + the deps for you:

curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env    # one-time: install uv

# serve — uv fetches Python 3.12 + mlx-vlm on first run · MLXFP4 (FP4, compact)
uv run --python 3.12 --with mlx-vlm -- \
  python -m mlx_vlm.server --model AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLXFP4 --port 8080 --max-kv-size 16384

Call it like an OpenAI endpoint (POST http://localhost:8080/v1/chat/completions) with the request "model" set to the launched id. (While this repo is private, run hf auth login first — or pass a local --model path.)

Sampling — set temperature: 1.0. The MLX server defaults to greedy decoding (temperature 0), which can repeat or loop on long prompts. This model is tuned for its native sampling — temperature 1.0 (top_p 0.95, top_k 64). Pass it in every request (clients that send no sampling params fall back to greedy):

curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLXFP4","messages":[{"role":"user","content":"Hello!"}],"temperature":1.0}'

Full multimodal is on by default (no flag) — send OpenAI image_url or input_audio content, or use mlx_vlm.generate --image pic.jpg / --audio clip.wav. Verified: describes images and transcribes speech.

Already have Python 3.12? Use a venv instead (+ one-off generate)

python3 -m venv .venv && source .venv/bin/activate
pip install -U mlx-vlm
python -m mlx_vlm.server   --model AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLXFP4 --port 8080
python -m mlx_vlm.generate --model AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLXFP4 \
  --prompt "Explain mixed-precision quantization." --max-tokens 512 --temperature 1.0   # add --image pic.jpg for vision

⚡⚡ Optional second deployment — +MTP speculative decoding (~1.1–1.2× faster, output-identical)

Google ships an official Gemma-4 MTP draft — google/gemma-4-12B-it-assistant (423M), an "assistant" head that proposes tokens this model then verifies. Because every token is verified, the output is identical — purely a throughput boost. The server auto-pulls the latest draft on first run (gated → run hf auth login once). Use --draft-block-size 2 — the benchmarked sweet spot on this quant; drafting deeper is slower (draft acceptance decays with depth on abliterated/quantized weights).

To run MLXFP4 + MTP, paste this into your terminal:

uv run --python 3.12 --with mlx-vlm -- python -m mlx_vlm.server \
  --model AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLXFP4 --port 8080 --max-kv-size 16384 \
  --draft-model google/gemma-4-12B-it-assistant --draft-kind mtp --draft-block-size 2

Pre-fetch/refresh the draft explicitly with hf download google/gemma-4-12B-it-assistant. Lossless; +~0.9 GB RAM. Remove the three --draft-* flags to disable. (Measured ~1.1–1.2× on this abliterated build; the draft is tuned for stock Gemma-4, so stock targets see more.)

Want maximum fidelity? The near-lossless MLX-8bit (FP8) build is the sibling.

🖥️ Minimum specs & unified memory

Comfortable on 16 GB Macs (incl. MacBook Air / base M-series). Want maximum fidelity with 24 GB+? Use MLX-8bit.

MLX quant grid

Why mixed mxfp4 / mxfp8 — high quality at the smallest size

A naïve uniform 4-bit quant of an abliterated model is a trap. The K=4 biprojection edit (source card) works by orthogonalizing self_attn.o_proj and mlp.down_proj — the residual-stream writers — against a 4-D refusal subspace on 24 of 48 layers. Round-to-nearest 4-bit noise re-corrupts that subspace, which on abliterated Gemma-4 lets refusals creep back and triggers a repetition-loop collapse after ~200–300 tokens.

So this build keeps the two abliteration writers (o_proj, down_proj) at 8-bit mxfp8, the soft-capped tied head and the (encoder-free) vision/audio projectors at bf16, and 4-bits only the quant-tolerant, residual-reading matrices.

How it compares: MLX's mxfp4 is E2M1 (one mantissa bit) — coarser than NVIDIA's NVFP4 — so we tune this build for size + speed while staying high-quality. On the BF16 model's own greedy trajectory it reproduces BF16's next token 88.5% of the time with median KL ≈ 0.004 nats — near-lossless on the typical token, with most divergence confined to genuinely ambiguous positions where multiple continuations are valid. The result reads as fully coherent, abliteration-intact Gemma-4 at just 9.3 GB. For the tightest possible match to BF16, the near-lossless MLX-8bit sibling (top-1 0.924) is the companion build.

Precision map

192 mxfp4 + 136 mxfp8 + 4 bf16-skip quantizable linears · 6.64 bits/weight.

Validation (MacBook Pro M4 Pro, 48 GB)

⏱️ Performance — measured on MacBook Pro M4 Pro · 48 GB

All figures below were benchmarked on a MacBook Pro · Apple M4 Pro (14-core CPU, 48 GB unified memory) · macOS 26 · mlx-vlm 0.6.1. Use them as a relative reference for your own Mac: a base M4 / M3 runs somewhat slower, an M4 Max / Ultra notably faster; MLX single-stream throughput is mostly memory-bandwidth bound. This compact build peaks ~10 GB, so it's comfortable on 16 GB Macs.

Greedy, post-warmup, median of 5 runs (benchmark.py).

Inherited K=4 abliteration capability vs google/gemma-4-12B-it: wikitext PPL drift −4.22%, HumanEval functional +6.7pp, IFEval 90% (see source card).

Quantization recipe

Built with mlx 0.31.2 / mlx-vlm 0.6.1 via a custom callable predicate to mlx_vlm.convert (the only route to mixed-mode FP4 — the stock --quant-predicate mixed_* recipes are affine-only and silently ignore --q-mode mxfp4):

from mlx_vlm import convert
SKIP      = ("embed_tokens", "lm_head", "vision_embedder", "embed_vision", "embed_audio")
PROTECT_8 = ("self_attn.o_proj", "mlp.down_proj", "self_attn.v_proj")
def pred(path, m):
    if not hasattr(m, "to_quantized"):  return False
    if any(s in path for s in SKIP):    return False
    if any(p in path for p in PROTECT_8): return {"group_size": 32, "bits": 8, "mode": "mxfp8"}
    return {"group_size": 32, "bits": 4, "mode": "mxfp4"}
convert(hf_path="<bf16>", mlx_path="<out>", quantize=True, q_mode="mxfp4",
        q_bits=4, q_group_size=32, dtype="bfloat16", trust_remote_code=True, quant_predicate=pred)

No calibration required (RTN); the recipe is the precision map.

Container & toolkit

ghcr.io/aeon-7/gemma4-aeon-abliterated-mlx-toolkit — the reproducible quant + validation + serve pipeline, an AGENTS.md agent-setup guide, and an elaborate model-comparison card. Quickstart is at the top of this page; on macOS run host-native for Metal (Docker has no Metal passthrough — see the toolkit's notes).

Technical details

Provenance

• Source (BF16): AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-BF16 — K=4 multi-direction norm-preserving biprojection; edits o_proj+down_proj on 24/48 layers (basis L24/L37/L39/L26).
• Original base: google/gemma-4-12B-it by Google.
• Quantized by AEON-7 on Apple Silicon (MacBook Pro M4 Pro, 48 GB) with mlx-vlm. Recipe designed + adversarially validated with AI-engineering assistance from Anthropic.

---

Arbitration Clause

By accessing, downloading, using, running inference on, fine-tuning, merging, quantizing, distributing, integrating, or otherwise interacting with this model, you acknowledge and agree to the following:

1. Sole Responsibility. You, the user, are solely and exclusively responsible for (a) every prompt you or your downstream system issue to this model, (b) every response this model produces in reply, (c) every downstream action taken by you, your systems, your agents, or your users in reliance on those responses, and (d) any harm — direct, indirect, consequential, foreseeable, or otherwise — that results from any of the above.

2. No Warranty. This model is provided strictly "AS IS", without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, non-infringement, safety, alignment, factual accuracy, or legal compliance in any jurisdiction. No contributor, author, publisher, or hosting platform assumes liability of any kind for outputs or downstream use.

3. Legal Compliance. You are responsible for ensuring that your use of this model complies with all applicable laws, regulations, terms of service, industry codes of conduct, professional ethical standards, and organizational policies in every jurisdiction in which you operate or in which your outputs may be received. The unaligned nature of this model does not grant you any legal authorization you did not already have.

4. Operational Safety Layer. An uncensored model is not a toy. You are expected to implement appropriate downstream safety layers proportionate to your deployment context, including but not limited to: input validation, output filtering, content moderation, audit logging, rate limiting, access controls, and human-in-the-loop review for high-risk workflows. A production deployment of this model without such layers is unsafe by construction and is not a supported use case.

5. Heightened Duty of Care. The absence of internal refusal behavior means the duty of care that would ordinarily rest partly with the model rests entirely with you. You are expected to exercise greater — not lesser — caution, forethought, and ethical discipline when operating this model than you would operate a base aligned model. If you are uncertain whether your contemplated use is ethical, legal, or wise, the correct action is to not make the request.

6. No Endorsement of Outputs. The authors, contributors, and publishers of this model do not endorse, adopt, or take responsibility for any specific output this model produces. Outputs are a stochastic function of the prompt, the weights, and the sampler state — not a statement of position by any human.

7. Arbitration. Any dispute, claim, or controversy arising out of or relating to the use of this model, its outputs, or this clause shall be resolved through binding individual arbitration under the rules of a mutually agreed arbitration body (or, absent agreement, the American Arbitration Association's Consumer Arbitration Rules), waiving any right to a jury trial, class action, representative action, or consolidated proceeding. Venue shall be the jurisdiction of the disputing party bringing the claim. Costs and attorneys' fees shall be allocated per the applicable arbitration rules. This clause does not expand, and where legally prohibited does not establish, any liability in the other direction; it limits how the user may proceed when alleging harm tied to their own use of this model.

8. Indemnification. You agree to indemnify, defend, and hold harmless the authors, contributors, and publishers of this model from and against any claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys' fees) arising from or related to your use of the model or your breach of this clause.

9. Severability. If any provision of this clause is held unenforceable in a given jurisdiction, the remaining provisions remain in full force in that jurisdiction, and the unenforceable provision is replaced by the closest enforceable equivalent consistent with the original intent.

10. Acceptance. Your use of this model constitutes your acceptance of this clause in full. If you do not accept, do not use the model.

This model is a tool with no opinions of its own. You supply the opinions. You supply the judgement. You supply the ethics. The outputs carry your fingerprints, not the model's.

---

License

Inherits the Gemma license from the base model. By using this model you agree to Google's Gemma license terms.

---

☕ Support the work

AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development:

Become a member on Patreon → patreon.com/cw/AeonForge7/membership

Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Supporters also get early access to new releases, such as the AEON DFlash2 drafter.

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.