gemma-4-E2B-it-txt-mlx-4bit

The text backbone of google/gemma-4-E2B-it, converted to MLX and quantized to 4-bit (group size 64, ~2.5 GB). It runs on its own as a small, fast chat model, but its main purpose here is to act as a speculative-decoding draft model for the 12B text models:

• jedisct1/gemma-4-12B-it-txt-mlx (bf16)
• jedisct1/gemma-4-12B-it-txt-mlx-8bit

E2B and the 12B share the exact same tokenizer and 262,144-token vocabulary (verified by hash), which is the requirement for using one as a draft for the other.

Why a draft model

Speculative decoding runs this small model to propose several tokens at a time, then lets the big model verify them in a single pass. Accepted tokens are free, so you get the big model's output at a fraction of the time. It is not MTP — it is the standard mlx_lm --draft-model mechanism, so it works with a stock mlx_lm install and nothing else.

Speeding up the 12B

mlx_lm.generate \
  --model jedisct1/gemma-4-12B-it-txt-mlx \
  --draft-model jedisct1/gemma-4-E2B-it-txt-mlx-4bit \
  --num-draft-tokens 6 \
  --prompt "Write a Python function that returns the nth Fibonacci number." --temp 0.0

Measured on an Apple Silicon machine (128 GB, temperature 0, 120 tokens — absolute numbers vary with load and thermals, but the ratios hold):

The win is largest with the bf16 target (the slow target benefits most). The 8-bit target is already fast, so the draft adds only a little there. --num-draft-tokens 4–6 is the sweet spot for this pair; higher values start to hurt when proposals get rejected.

A draft never changes correctness: tokens are only kept when they match what the 12B would have produced. Tool calls in particular come out identical to running the 12B alone — verified end to end with the Gemma 4 tool-call format. (At greedy temperature 0, very long generations may differ from the no-draft run by the occasional near-tie token, because the target verifies drafts in a batch; this is normal for MLX speculative decoding and does not affect quality or tool calling.)

Standalone use

mlx_lm.generate --model jedisct1/gemma-4-E2B-it-txt-mlx-4bit --prompt "Hello!" --temp 0.0

Conversion details

• Source: google/gemma-4-E2B-it (text backbone only; vision/audio dropped, model_type gemma4).
• Tool: mlx_lm.convert, mlx_lm 0.31.3 / mlx 0.31.2; 4-bit affine, group size 64.
• E2B uses KV-shared layers (num_kv_shared_layers=20); the unused shared-layer key/value projections in the checkpoint were dropped during conversion so the weights match the mlx_lm runtime.

License

Apache 2.0, same terms as the original Gemma 4 release.