Gemma 4 12B IT — TT-native all-BFP8, single P150, 262K context, speculative decoding and on-device sampling

A hardware-specific, text-only checkpoint and custom runtime derived from google/gemma-4-12B-it, revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7, for one Tenstorrent Blackhole P150. The drafter for speculative decoding is derived from google/gemma-4-12B-it-assistant, revision 46d4c6f13f0ac0ad827b915669b8df9b81c64c51.

This is a community experimental release by Lottolabs, not an official Google or Tenstorrent release. It is not GGUF, AWQ, GPTQ, bitsandbytes, or an ordinary Transformers checkpoint. transformers.from_pretrained(), llama.cpp and hosted Inference Providers cannot load it. It needs the bundled runtime.

Quantization

All-BFP8: every text matrix (attention, MLP, LM head) is stored as TTNN bfloat8_b. The embedding is BF16; 1-D tensors (norms, layer scalars) are lossless BF16 in small.safetensors. See gemma-4-12B-it/precision_plan.json and gemma-4-12B-it/native_manifest.json.

tensors/tensor_cache_bf16/*.tensorbin holds the exact TTNN host tensors that the runtime uploads to device DRAM. The directory name comes from the runtime's tensor-cache writer. The files are not a regenerable cache: they are the checkpoint. Every file is hashed in the manifest, and the loader refuses to start if any file is missing or changed. The drafter (gemma-4-12B-it-assistant/) stores its tensors the same way. Its recorded dtype is bfp8-lm4: BFP8 weights and a BFP4 drafter LM head.

The loader requires two proofs, both bound to the manifests and recorded on the current runtime (p3d-pc-gate-s1, see Runtime):

• gemma-4-12B-it/equivalence.json: the full logits of the native reload are bit-exact to quantizing the original weights at load time, over 4,093 teacher-forced tokens (exact_logits_equal: true).
• gemma-4-12B-it-assistant/spec_equivalence.json: greedy speculative decoding with draft length 5 produced token streams identical to non-speculative greedy decoding on the same proven runtime (streams_identical: true, 3,770 tokens compared, including 32K and 131K prompts that run the confidence-gated cycle).

Quality versus the original BF16 model

This table is the precision-selection study. It is a teacher-forced comparison against the original model run in BF16 on CPU, measured on the exact (unfused) TT build (evidence/quality/quant_table.json). For the shipped runtime's own gate, see Runtime. dPPL is the perplexity change relative to BF16, top-1 is argmax agreement with BF16, and KL is the mean KL divergence on the short-prompt set:

Why BFP8 and not BFP4: with the MLP in BFP4, perplexity rises about 5% and top-1 agreement falls to about 87%. All-BFP4 is worse still. BFP8 stays within about 1–2% of BF16 perplexity.

Runtime

The current runtime is p3d-pc-gate-s1: image lottolabs/gemma4-12b-tt-p150:p3d-pc-gate-s1, ID sha256:7447e3f5a1f1bd9f83cb60c3d4d22cd7fa3d44195b31a53982c63b988e660793. Its Gemma 4 tree and vllm_tt_plugin package are at git commit d6f9202. It is the p3d runtime (same numerics switches, same checkpoint tensors) plus four serving features:

1. Exact prefix caching (on by default). A prompt that extends an earlier prompt resumes at the last 16,384-token chunk boundary the two share, and then runs the same chunks a full prefill would run. Logits, full-attention KV, ring windows and token streams were checked bit-identical to a full prefill at resume points from 16K to 64K (evidence/p3d-pc-gate-s1/prefix-cache/resume-check/). Up to 4 resume snapshots (about 0.34 GB of host RAM each) are kept, least recently used first out.
2. Confidence-gated draft length at long context. From 4,096 tokens of context the drafter stops drafting when its top probability falls below 0.85, and from 49,152 tokens below 0.95. Below 4,096 tokens the cycle is the fixed K=5 cycle of p3d. Greedy output is unchanged by the gate (the speculative proof covers gated 32K and 131K prompts).
3. On-device exact sampling. Temperature, top-k (1–64), top-p, min-p and presence/frequency/repetition penalties run on device with a seeded Philox RNG and reproduce vLLM's sampler. Requests that use logprobs, allowed_token_ids, bad_words, logit_bias, an unsatisfied min_tokens, structured outputs or a top-k outside 1–64 fall back to vLLM's exact host sampler.
4. Lossless speculative sampling. Sampled requests now decode speculatively too: the drafter proposes up to 5 tokens and the target accepts or resamples them by standard speculative rejection sampling, so the output distribution is the target model's. On p3d, sampled requests decoded one token per step on the host sampler.

The release also fixes the drafter reading a row of the target KV cache that the current request had not yet written (the first cycle after prefill, and after a fully accepted cycle). That row could hold data from an earlier request. Greedy output was never affected, because drafts only change acceptance there, but in sampled mode it made seeded outputs depend on earlier requests. The device test for this failed before the fix and passes after it (evidence/p3d-pc-gate-s1/sampling/seedtest-g1.log, seedtest-g2.log).

The numerics switches are the same as p3d's, and the proofs bind them (runtime_env: GEMMA4_VERIFY_SDPA=batched, GEMMA4_FUSE_GELU_MUL=1, GEMMA4_DECODE_KV_ROWS=1, GEMMA4_DECODE_MM_DN=1). Greedy outputs on the recorded exact prompts are token-identical to p3d's (3,130 of 3,130 tokens, evidence/p3d-pc-gate-s1/sampling/runs/ref-vs-shipped.json). The checkpoint tensors are byte-identical to the p3d release; only equivalence.json and spec_equivalence.json were re-recorded on this runtime.

Quality gate for the p3d numerics (unchanged in this runtime). Decode-path teacher forcing was compared against the original model in BF16 on CPU (evidence/quality/p3d-gate/):

Previous releases

• p3d (runtime lottolabs/gemma4-12b-tt-p150:p3d, ID sha256:331e79b24fa2575c436158be994c6bc0e184d776d15305e82538d2e4d646f544; greedy-only speculation, host sampler, no prefix caching) is tag p3d, commit 1f97e442a1fc4f57e56aeace96048a5cfd8a89c8. Its evidence is under evidence/previous-p3d/. To serve it, fetch that revision's launcher and pass the revision; each launcher only accepts its own pinned runtime:

  curl --fail --location --output launch-p3d.py \
    https://huggingface.co/Lottolabs/gemma-4-12B-it-TT-BFP8-P150/resolve/1f97e442a1fc4f57e56aeace96048a5cfd8a89c8/launch.py
  python3 launch-p3d.py --revision 1f97e442a1fc4f57e56aeace96048a5cfd8a89c8 --cache-root /large-disk/gemma4-12b --device-ownership-confirmed
  

• p3c was published at revision 3d9001d5; its evidence is under evidence/previous-p3c/.

Performance

Measured on runtime p3d-pc-gate-s1 over HTTP on one P150: one user, speculative decoding on, 512-token outputs (ignore_eos), Gemma chat prompts, p50 over 3 requests (2 at 32K, 1 at 262K) with distinct seeds (evidence/p3d-pc-gate-s1/sampling/summary-p3d-pc-gate-s1.md, summary.json). Decode tok/s:

A request that leaves every sampling field at its default (the checkpoint's generation_config.json: temperature 1.0, top-p 0.95, top-k 64) decoded at 54.7 tok/s at 128 tokens, against 17.7 tok/s on p3d. Greedy speed is unchanged from p3d (55.97 vs 55.96 tok/s at 128 tokens, 56.7 vs 57.1 at 2K, on the same probe). Without speculation (--speculation off), sampled decode on device runs at 26.2 / 25.5 / 24.6 / 19.6 tok/s at the four prompt lengths.

TTFT is 0.1 s at 128 tokens, 0.65 s at 2K, 17 s at 32K and about 6.5 min at 262K on a cold prompt. Prefix caching was measured when it was added (layer p3d-pc, the p3d runtime plus prefix caching): a second turn that extends a 32K, 64K or 128K conversation started its answer in 10.7 / 15.6 / 25.3 s instead of 18.7 / 45.3 / 126.3 s, and repeating that turn took 0.7 / 1.1 / 1.7 s; all texts equal the run without prefix caching (evidence/p3d-pc-gate-s1/prefix-cache/http/). Prompts shorter than 16,384 tokens have no resume point and prefill in full.

When it was added (layer p3d-pc-gate), the draft-length gate raised greedy HTTP decode at 32K from 43.2 to 47.6 tok/s and at 261,632 tokens from 24.4 to 30.6 tok/s on the chat throughput probe, with 128 and 2K unchanged and TTFT unchanged (evidence/p3d-pc-gate-s1/gating/p3d-pc-gate/).

Speculative decode speed depends on how many drafted tokens the content lets the model accept, which is why it does not fall monotonically with prompt length.

LocalMaxxing, 2K prompt (p3d, greedy). The p3d server reached 53.9 output tok/s (median of 5), 3,063 tok/s prefill and a 651 ms TTFT, with 1,996 prompt and 256 output tokens. The run was submitted as cmuosddj10kd1lq010lf2ahkj and approved as an unverified run (evidence/localmaxxing/p3d-2k/speed-test.json). Greedy speed is unchanged in this runtime.

Serving correctness and sampling quality (p3d-pc-gate-s1)

• Greedy HTTP output matches the direct runtime token for token, with and without speculation: 28 of 28 checks passed in each mode (evidence/p3d-pc-gate-s1/sampling/runs/spec-a/exact.json, nospec-a/exact.json). They cover chat and completion outputs on 10 prompts, request isolation, sampled requests followed by greedy ones, cancellation mid-prefill and mid-decode, and rejection of image input.
• Sampled output has the model's distribution. 250 samples per preset (recommended, coding, penalized) and prompt, with independent seeds, were compared by two-sample chi-square tests on the first four tokens and the length, 30 tests per pair. The smallest p-values were 0.0377 (speculative vs non-speculative device sampling), 0.0042 (speculative device vs the p3d host sampler) and 0.0163 (non-speculative device vs host sampler), all above the Bonferroni threshold 0.05/30 = 0.0017 (evidence/p3d-pc-gate-s1/sampling/dist/).
• Seeds reproduce. The same seed gives the same text, including after other requests, seeds, presets and prompts; different seeds give different texts (evidence/p3d-pc-gate-s1/sampling/runs/spec-a/smoke.json, seedtest-g2.log).
• GSM8K-100 with sampling (one 100-question run per seed): recommended preset with speculation 97 / 96 / 97 (non-speculative device sampling 97 / 98 / 96, p3d host sampler 95 / 98 / 97); coding preset with speculation 97 / 96 / 98 (p3d host sampler 94 / 95 / 96). A run took about 330 s with speculation and about 1,630 s on the p3d host path (evidence/p3d-pc-gate-s1/sampling/runs/spec-gsm/).
• Request parameters (37 cases per mode, param_matrix.json): bad_words, allowed_token_ids, logit_bias and logprobs are applied. With speculation on, upstream vLLM's speculative-decoding validation rejects min_p, logit_bias and min_tokens with HTTP 400; they work with --speculation off.
• Long context works to the native limit (measured on p3d; the long-context path is unchanged apart from the gate): passkeys at the beginning, middle and end of the prompt were retrieved at 32,768, 131,072 and 262,016 prompt tokens, and a request totalling 262,145 tokens was rejected with HTTP 400 (evidence/previous-p3d/http-p3d/long.json). With prefix caching on (layer p3d-pc), the 262,016-token passkeys were retrieved again and the overflow request was rejected (evidence/p3d-pc-gate-s1/prefix-cache/http/http-on/long.json).
• The public package was verified end to end. Revision 4bd4ffd2 was downloaded anonymously into an empty directory with its own launch.py --download-only (599 files, 21.3 GB, every file re-hashed against SHA256SUMS, 0 disagreements). With the local image tag removed, the launcher loaded the downloaded runtime archive and got the pinned image ID. Served from the download with fresh kernel caches, greedy HTTP output matched the direct runtime (28 of 28 checks), greedy decode ran at 56.0 / 56.8 tok/s and recommended sampling at 56.4 / 52.3 tok/s at 128 / 2K prompt tokens, and seeded requests reproduced (evidence/public-download-verification.json, evidence/public-serving-smoke.json).

Download and serve

Prerequisites:

• Linux with a supported P150 driver and hugepage mounts at /dev/hugepages and /dev/hugepages-1G.
• Docker and Python 3.11 or newer.
• Exclusive ownership of one P150.
• Disk space for about 21.3 GB of downloads: a 14.7 GB checkpoint, a 1.2 GB drafter and a 5.4 GB compressed runtime archive. You also need room for the loaded image and a writable kernel cache.

Download the standard-library launcher:

curl --fail --location --output launch.py \
  https://huggingface.co/Lottolabs/gemma-4-12B-it-TT-BFP8-P150/resolve/main/launch.py

python3 launch.py \
  --cache-root "$HOME/.cache/gemma4-12b-tt-native" \
  --device-ownership-confirmed

The launcher does the following before it serves anything:

1. Resolves the requested Hugging Face revision to an immutable commit.
2. Downloads only the serving package (the checkpoint, drafter, server script and runtime archive) and verifies every byte against runtime-release.json.
3. Checks that the published file sets match both native manifests.
4. Loads the checksum-pinned runtime archive into Docker and checks the resulting image ID.

serve_native.py then verifies the manifests, file hashes and both proofs again, checks the image ID once more, and starts vLLM. Inside the container, the loader repeats the checks against the image's runtime identity.

The original BF16 weights are not needed. The API is OpenAI-compatible, binds to 127.0.0.1:8000, and serves the model name google/gemma-4-12B-it. The first launch compiles kernels into the kernel cache, so it takes longer than later launches.

Useful modes:

# Download and verify without Docker or a device
python3 launch.py --cache-root /large-disk/gemma4-12b --download-only

# Print the exact serving command without starting it
python3 launch.py --cache-root /large-disk/gemma4-12b --print-command

# Non-speculative serving (no drafter)
python3 launch.py --cache-root /large-disk/gemma4-12b --speculation off --device-ownership-confirmed

# Without prefix caching
python3 launch.py --cache-root /large-disk/gemma4-12b --prefix-caching off --device-ownership-confirmed

Save the immutable revision that the launcher prints, and pass it with --revision to reproduce the same setup exactly.

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "google/gemma-4-12B-it", "temperature": 0, "max_tokens": 256,
  "messages": [{"role": "user", "content": "Explain database transactions in three sentences."}]}'

# Sampled with Google's recommended settings, reproducible with a seed
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "google/gemma-4-12B-it", "temperature": 1.0, "top_p": 0.95, "top_k": 64, "seed": 7, "max_tokens": 256,
  "messages": [{"role": "user", "content": "Write a haiku about lighthouses."}]}'

Limits

• One user at a time (max_num_seqs 1), one P150 (MeshShape 1x1), no multi-device support.
• Text only. Requests with images are rejected.
• Context up to 262,144 tokens in total. Sliding-window layers keep their KV cache as 18,432-token rings. Prefill runs in 16,384-token chunks; prefix caching resumes only at those chunk boundaries, so prompts shorter than 16,384 tokens are always prefilled in full.
• With speculation on, min_p, logit_bias and min_tokens are rejected with HTTP 400 by upstream vLLM's speculative-decoding validation. Serve with --speculation off to use them.
• Requests that need logprobs, allowed_token_ids, bad_words, structured outputs or a top-k outside 1–64 (including top-k disabled) are sampled on the host: exact, but one token per step and at about the p3d host speed.
• Long prompts have a long time to first token on a cold prompt: about 17 s at 32K and about 6.5 min at 262K.

Integrity and provenance

• runtime-release.json: the serving inventory the launcher downloads, the runtime archive checksum, the expected Docker image ID and the upstream revisions.

• release-manifest.json: the complete repository inventory with size and sha256 for every file.

• SHA256SUMS: file checksums.

• reproduction.json: source, runtime, serving and evidence contract.

• runtime/: source of the runtime image layers, recorded in runtime/source.json. It contains:

  • the Gemma 4 TT model tree (runtime/gemma4/, git commit d6f920246c37a9b6eeed74b6bd853da683e4ff7b, copied from the image);
  • the TTNN overlay (runtime/ttnn-overlay/: 1D matmul factory and the dual-NoC in1 reader, dn commit efc465f4b1a44025ae4a33e4dbc7d371b3f9266c, byte-identical to the image);
  • the vLLM core files and the complete vllm_tt_plugin package as installed in the image (runtime/serving-overlay/);
  • the Dockerfile chain (p2b → p2c → ttnn-dn1 → fast2 → p3d → p3d-pc → p3d-pc-gate → p3d-pc-gate-s1) and the build script;
  • the sampling patches (evidence/p3d-pc-gate-s1/sampling/source/).

  This is provenance: serving uses the checksum-pinned runtime-image.tar.gz.

• native_checkpoint.py and provenance/build_native_checkpoint.py: the manifest, proof and verification tool, and the checkpoint builder.

License and attribution

Derived from google/gemma-4-12B-it and google/gemma-4-12B-it-assistant. Apache-2.0; see LICENSE. The runtime includes tt-metal and vLLM, which keep their upstream notices; see runtime/licenses/.