Sign up

litert-community/granite-4.0-350m-litert-lm

🤗 On Hugging Facetext-generationapache-2.0936 MBother⚠ No SHA-256 weightsupdated today
Magnet

granite-4.0-350m-litert-lm

ibm-granite/granite-4.0-350m converted to the LiteRT-LM

.litertlm format for on-device inference with

LiteRT-LM (Android, iOS, desktop, embedded).

  • Base model: ibm-granite/granite-4.0-350m
  • License: apache-2.0 (inherited from the base model)
  • Format: .litertlm (granite-4.0-350m_q8_ekv1280.litertlm)
  • Quantization: 8-bit dynamic — weights int8, activations fp32 (dynamic_wi8_afp32)
  • KV-cache / max context: 1280 tokens

Usage

# CLI
litert-lm run granite-4.0-350m_q8_ekv1280.litertlm --prompt "What is the capital of France?"

The bundle embeds the base model's tokenizer and chat template, so prompts are

templated for you. It also serves over an OpenAI-compatible endpoint:

litert-lm serve granite-4.0-350m_q8_ekv1280.litertlm --host 127.0.0.1 --port 9379
# then POST /v1/chat/completions with "model": "granite-4.0-350m_q8_ekv1280"

Conversion recipe

Converted with Google's litert-torch

generic HF export (litert-torch==0.9.1, generative.export_hf): the checkpoint's own

transformers modeling code is torch.exported (prefill + decode, static KV cache),

quantized with the AI Edge Quantizer, and bundled with the HF tokenizer + chat template —

no per-family builder.

A lossless dense reinterpretation is applied first (all-attention GraniteMoeHybridForCausalLM → dense GraniteForCausalLM), because the hybrid wrapper is not torch.export-compatible against a static cache (see litert-torch#933).

Validation / Parity

This conversion ships recorded parity evidence against the Hugging Face reference (generated by the workbench parity gate; see parity_report.json).

Weight-remap parity (argmax-identical logits, torch↔torch)

Granite Nano ships under the GraniteMoeHybridForCausalLM class but is all-attention; it is losslessly reinterpreted as a dense GraniteForCausalLM before export. Logits of the original vs. the reinterpreted model over the pinned prompts:

  • argmax match: 100.0% over 5 prompts — greedy decoding is unaffected by the remap
  • max abs logit diff: 7.41e-05 (within 0.001; residual is fp round-off from splitting the fused SwiGLU)

Runtime parity (HF reference vs .litertlm, greedy)

Greedy decoding (temp 0 / top-k 1) through each model's own chat template, over 5 pinned prompts:

  • exact text match: 20.0%
  • reference contained in device output: 20.0%
Honest caveat: litert-lm exposes no logits and the export is quantized to int8, so runtime parity is text/token parity, not logit parity. It is token-exact on short prompts; longer generations may diverge due to int8 + greedy argmax sensitivity — expected, not a conversion bug.

Limitations

  • Quantized (q8): on-device outputs may differ slightly from the fp32 reference,

especially on longer greedy generations.

  • CPU backend validated; other backends untested here.

Training data

Format conversion only — no additional training data was used. Training-data composition and governance are inherited from the base model, ibm-granite/granite-4.0-350m (see its model card).

All personally identifiable information has been scrubbed from the training data.

Provenance

  • Base: /workspace/models/ibm-granite_granite-4.0-350m
  • Toolchain: litert-torch==0.9.1
  • sha256: c8e9a29493f62b7c44461fb36980987c4c1454c75e95f57ba0539a8edc9dce76
  • Converted and parity-validated with the edge-workbench pipeline.

Acknowledgments

Base model by its original authors (see the base model card). Conversion tooling:

Google AI Edge (LiteRT-LM, litert-torch).