gemma-4-12B-it-txt-mlx-8bit

This is the text backbone of google/gemma-4-12B-it converted to MLX for Apple Silicon and quantized to 8-bit (group size 64, ~8.5 bits per weight, ~12 GB on disk). It is the near-lossless companion to the full-precision jedisct1/gemma-4-12B-it-txt-mlx.

Gemma 4 12B is a gemma4_unified any-to-any model: text, vision and audio in, text out. This conversion keeps the language model only — the vision and audio towers are dropped, which is why the name carries a txt marker. If you need image or audio understanding, use the original model with a multimodal runtime. If you want a fast, local, text-and-tool-calling model that runs on stock mlx_lm, this is it.

What works

• Runs directly with the standard mlx_lm CLI and server — nothing custom to install.
• Tool calling is reliable. Gemma 4 emits calls in its own delimiter-based format (<|tool_call>call:name{arg:<|"|>value<|"|>}<tool_call|>); mlx_lm's built-in gemma4 tool parser turns that back into standard OpenAI tool_calls JSON, so OpenAI-compatible clients work out of the box.
• Reasoning ("thinking") is supported. mlx_lm turns it on by default for this model; see below if you want it off.

Usage

mlx_lm.generate --model jedisct1/gemma-4-12B-it-txt-mlx-8bit \
  --prompt "Explain the Monty Hall problem in three sentences." --temp 0.0

From Python:

from mlx_lm import load, generate

model, tokenizer = load("jedisct1/gemma-4-12B-it-txt-mlx-8bit")
messages = [{"role": "user", "content": "Write a haiku about pointers."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

Tool calling and OpenAI-compatible serving

mlx_lm.server --model jedisct1/gemma-4-12B-it-txt-mlx-8bit --port 8080

Call it like any OpenAI chat endpoint with a tools array; the response comes back with finish_reason: "tool_calls" and a normal tool_calls list:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "jedisct1/gemma-4-12B-it-txt-mlx-8bit",
  "messages": [{"role": "user", "content": "What is the weather in Paris in celsius?"}],
  "tools": [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city.",
    "parameters": {"type": "object",
      "properties": {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}},
      "required": ["city"]}}}],
  "temperature": 0.0
}'

This model has been exercised end-to-end as the backend for the swival coding agent and drives its full tool loop. For agentic and coding work, use temperature 0.0 — greedy decoding is markedly more reliable for tool arguments and code than the model's creative defaults (temperature 1.0, top_k 64, top_p 0.95).

Thinking on/off

mlx_lm enables Gemma 4's thinking channel by default. To turn it off:

mlx_lm.server --model jedisct1/gemma-4-12B-it-txt-mlx-8bit --chat-template-args '{"enable_thinking": false}'

or --chat-template-config '{"enable_thinking": false}' with mlx_lm.generate. Tool calling is correct either way.

Faster generation with a draft model

You can pair this with the small E2B draft (same tokenizer/vocab) for speculative decoding via stock mlx_lm:

mlx_lm.generate \
  --model jedisct1/gemma-4-12B-it-txt-mlx-8bit \
  --draft-model jedisct1/gemma-4-E2B-it-txt-mlx-4bit \
  --num-draft-tokens 4 \
  --prompt "Explain speculative decoding in two sentences." --temp 0.0

The 8-bit target is already fast, so the gain here is modest (the bf16 model benefits far more); try --num-draft-tokens 3–4 and keep it only if it's faster on your machine. Output and tool calls are unchanged. Drafts: gemma-4-E2B-it-txt-mlx-4bit, gemma-4-E2B-it-txt-mlx-8bit.

Conversion details

• Source: google/gemma-4-12B-it.
• Tool: mlx_lm.convert, mlx_lm 0.31.3 / mlx 0.31.2.
• Quantization: 8-bit affine, group size 64 (~8.5 bits per weight).
• Vision and audio weights (vision_embedder, embed_vision, embed_audio) were removed; only the language model is kept. model_type set to gemma4.

License

Released under the Apache 2.0 license, the same terms as the original Gemma 4 release. Please review Google's terms before use.