A fitted state bridge: Granite 4.0 H 350M reads the prompt, and this bridge translates what it built into the memory Granite 4.0 H Small needs. H Small is a 32B mixture-of-experts with 72 experts, about 95 times larger than the 350M. It then answers from that memory, reading one token of the prompt itself.
It applies LatentPort, Simon Villani's method (arXiv:2609.25053), to a model whose memory is mostly Mamba-2 recurrent state. The reproduction it belongs to is PrattleNot.
It is for H Small quantized to Q4K. The bridge was fitted against Granite 4.0 H Small with its linear layers quantized to Q4K when it loads (PrattleNot's
PRATTLE_TARGET_DTYPE=q4k), and must be used with H Small loaded the same way.
⚠️ Security note. When bridged, the answering model does not read the prompt itself, the system prompt included, and the bridge does not reliably carry its instructions. Measured on six of our pairs, a rule set in a system prompt is broken more often bridged than when the large model reads the prompt. Do not rely on system-prompt instructions as guardrails for a bridged model: have the large model read the system prompt itself, or enforce policy outside the model.
H Small has 40 layers: 36 Mamba-2 layers and 4 attention layers (5, 15, 25, 35). The 350M has 32, and each H Small layer reads the 350M layer at the same relative depth.
Held-out questions, never fitted on:
| share of the read-nothing → read-everything gap the bridge closes (NCR) | the same, from a wrong donor | correct of 7 | |
|---|---|---|---|
| the maps alone — this file | 0.767 | 0.161 | 7; H Small alone: 7, the 350M alone: 5 |
| plus the paper's learned correction — not published | 0.826 | — | 6 |
NCR is the paper's native-closure ratio (arXiv:2609.25053, Eq. 8):
(ℓ_empty − ℓ_bridged) / (ℓ_empty − ℓ_native), where each ℓ is H Small's loss on its own native
reply, scored teacher-forced. It is a loss ratio, not a rate of matching tokens. Per token, H Small's
loss on its own reply is 0.204 reading everything, 0.668 bridged, 2.195 reading nothing, and 1.874
from a wrong question's memory.
Why the corrected bridge is not the one published. The paper's learned correction lifts NCR to 0.826, but the corrected bridge answers 6 of 7 held-out questions. Our bar is that the bridged model answers every question the target answers alone, and H Small alone answers 7. So the file here is the uncorrected maps, which answer all 7.
Two runs of the same command:
| prompt | run | time to first token, bridged H Small vs H Small reading the prompt | the ceiling in the same run (H Small alone ÷ 350M alone) |
|---|---|---|---|
| ~256 tokens | 1 | 1.69× sooner | 6.45× |
| ~256 tokens | 2 | 1.75× (1.74–1.75) sooner | 7.30× |
| ~1024 tokens | 1 | 1.61× sooner | 7.39× |
| ~1024 tokens | 2 | 1.62× (1.61–1.62) sooner | 7.58× |
Measured on two RTX 3090s, the 350M in bf16 with the bridge on one and H Small quantized to Q4K on the other: synthetic prompts, 8 prompts × 3 repetitions in rotated order, one bridge token, and every run's counters checked, so the bridged H Small read no prompt history itself. The figure is the geometric mean of native ÷ bridged with its 95% interval, LatentPort's own statistic.
Why it stays far below the ceiling. At the handoff H Small must run its own projection, convolution and scan over 36 layers of translated input to rebuild its Mamba-2 state, about 260 ms of the bridged model's ~357 ms at ~256 tokens. A plain key/value bridge, such as our Qwen3-0.6B → 32B, has no such step, and gains more.
To repeat it from a PrattleNot checkout:
PRATTLE_SOURCE_DTYPE=bf16 PRATTLE_TARGET_DTYPE=q4k PRATTLE_DEVICES=cuda:0,cuda:1 just speedup models/granite-4.0-h-350m models/granite-4.0-h-small models/bridges/granite-4.0-h-350m-to-h-small.safetensors 256
Pass 1024 as the last argument for the longer prompts.
3b17b717b8f2f5d305b0a92c1491e239aeda19c8 and
ibm-granite/granite-4.0-h-small at
b8c0982bab7fde4eb48110f5a069527c008fab39, both Apache-2.0. Hidden width 768 and 4096; KV heads
4 and 8, both of width 128; H Small routes each token to 10 of its 72 experts.granite-4.0-h-350m-to-h-small.safetensors, 478 MB, with no __metadata__:
| tensors | what they hold |
|---|---|
k.<l> for a Mamba-2 layer (f16, 1537 × 4096) |
the map from the 350M's residual stream to H Small's normed input rows for that layer |
k.<l>, v.<l> for an attention layer (f16, 1537 × 1024) |
the map to H Small's keys / values |
layer_of |
which 350M layer feeds each H Small layer |
meta |
the shape constants the reader needs |
It is read by prattle_core::bridge in PrattleNot, in Rust. It is not a transformers model, and it
contains no weights of either Granite model. Fetch those from
IBM Granite.
PrattleNot is a derivative work of LatentPort by Simon Villani: https://latentport.com/, https://github.com/LatentPort/latentport, licensed under the Apache License, Version 2.0. The bridge design and the one-token handoff are the paper's (arXiv:2609.25053v1); translating a recurrent layer's inputs rather than its state is ours. Granite 4.0 is by IBM, under Apache-2.0. This bridge is released under Apache-2.0.
ibm-granite/granite-4.0-h-350m