NVIDIA-Nemotron-3.5-Bolt-E4B — LiteRT-LM

This repository is a conversion of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 to the LiteRT-LM (.litertlm) format, made with litert-torch's export_hf (0.10.0) for LiteRT-LM v0.18.0: on a Galaxy S26 GPU the int4 file prefills 1,024 tokens at 868 tokens/s and decodes at 15.2 tokens/s (2026-10-26). The phone lowered the GPU's clock during each of these runs, so the numbers are sustained values, not peaks. A second file is compiled for the same phone's NPU (Snapdragon SM8850) and decodes at 12.2 tokens/s there.

NVIDIA describes Bolt-E4B as "a small language model (SLM) further post-trained from Gemma-4-E4B-it by NVIDIA" that "further improves multi-turn tool use, instruction following, and input grounding for fast local voice/device assistants, interactive personas, and embedded/edge interfaces". Its card lists text in and text out and states that the model "is ready for commercial or non‑commercial use". The checkpoint also carries Gemma 4's image and audio encoders; these files leave them out.

The model is Gemma 4 E4B: 42 decoder layers (35 with a 512-token sliding window, 7 with full attention), hidden size 2560, 8 query heads and 2 key-value heads, a 262,144-token vocabulary plus a second embedding table that gives every layer its own 256 values per token, and the last 18 layers reading the keys and values of earlier layers.

Accuracy

The first questions of the GSM8K test split, greedy, each asked in its own conversation with a prompt that ends in "write the final answer on its own line in the exact form: #### ", up to 2,048 new tokens less the prompt. The two phone runs stopped when their time on the device ran out: 52 questions on the GPU, which heat had slowed to about a minute per question, and 99 on the NPU. Question by question, neither file differs from the reference by more than chance: exact McNemar p = 1.0 for the Mac GPU (6 questions right only in the reference, 5 only in the int4 file), the Galaxy S26 GPU (3 and 2) and the NPU (3 and 3).

The Galaxy S26 GPU questions ran on the int4 file before its 512-token prefill signature was added. These prompts are under 200 tokens and run through the 128-token prefill and the decode graphs, which the addition left unchanged; on the Mac GPU the file gave the same answer text as before on all 100 questions.

Eight short questions (a sum, a product, a capital, an opposite, the days in a week, a translation, a comparison of two decimals, a rhyme) got the right answer from the BF16 reference, from the int4 file on the Mac CPU and GPU and the Galaxy S26 GPU (on the Galaxy S26 CPU we asked the first question only; all eight were right there with the file before its 512-token signature was added), and from the NPU file on the Galaxy S26 NPU. Two questions asked at the end of a 444- and a 450-token prompt were answered correctly by both files on the Galaxy S26.

Usage

pip install litert-lm==0.18.0
litert-lm run ./NVIDIA-Nemotron-3.5-Bolt-E4B_int4.litertlm --backend gpu --cache no --prompt "Extract the dates from: The meeting moved from May 3 to May 17."

On a Mac M4 Max (litert-lm 0.18.0) this printed May 3, May 17 in 2.4 s including engine creation, at a peak memory footprint of 4.3 GB.

Thinking is off by default. --thinking true turns it on (--thinking-budget caps it): NVIDIA's template then starts the system turn with <|think|>, and the model may open a thought, which the files declare as a channel (<|channel>thought … <channel|>), so the runtime hands the thought back separately from the answer. A prompt, its thought and its answer share the 2,048-token cache.

The NPU file runs with LiteRT-LM's NPU executor and Qualcomm's QNN libraries; we ran it with a LiteRT-LM build from the main branch (commit 3dbb23e1, 2026-10-02, not in a release up to v0.18.1), the QNN libraries of QAIRT 2.47.0, and --use_hw_cache_update_for_npu=true. Whether released LiteRT-LM builds run it was not tested.

Performance

A forced 1,024-token prefill, then 256 decoded tokens, two or more runs per row. On the Galaxy S26 every run in the table started with the GPU and CPU at their top clock limits, and the phone lowered them during the run: in the GPU runs the GPU's limit fell from 1,300 MHz to between 578 and 1,100 MHz, and the CPU cores were capped in the GPU, NPU and CPU runs alike. The numbers are what the phone sustained, not its peak.

Phone rows are medians of runs started at the top clocks; memory is the higher run's peak resident set and the GPU driver's allocation (kgsl). A 512-token prompt runs through the 512-token signature: its first token came 1.40× sooner on the Galaxy S26 GPU (0.87 s; one of these two runs started with the phone's big CPU cluster already capped) and 1.79× sooner on the Mac GPU (0.21 s) than on the earlier build of the file without the 512-token signature. On the Galaxy S26 GPU, fp32 activations (--force_f32=true) decoded at 0.78× and prefilled at 0.18× the fp16 speed (measured on the file before its 512-token signature was added; its 1,024-token prefill and decode graphs are the same).

On a Pixel 8a (8 GB) we ran only earlier builds of the int4 file (without the 512-token signature), not the shipped one. Our test stops a run at the first reading of available memory under 400,000 KiB; every GPU run of a 1,024-token prefill reached that point (for example 348,756 KiB), so there are no Pixel 8a GPU numbers. On its GPU the int4 weights still answered the eight short questions with the Mac's answers (one of those runs was stopped by the same test after its answer).

Conversion notes

• Source: nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 at revision 49843eed, every file but the two README images checked against the Hub's record. model-frozen-shared-kv.safetensors holds the key and value projections and key norms of layers 24 to 41; those layers read the keys and values of earlier layers, so the files do not use these weights (they equal Google's Gemma 4 E4B weights byte for byte).
• int4 file: litert-torch export_hf 0.10.0 (0a81bf6f) with transformers 5.19.0; prefill signatures of 128, 512 and 1,024 tokens and one decode signature; --use_rope_composite=True (the rotary embedding as odml.rope composites); --apply_gpu_composites=True with its other GPU options off (fused projections, SwiGLU, short-conv, query-key-norm-rope and SDPA composites, the boolean mask, the dynamic cache), which writes the KV-cache updates and the attention matrix products as composites; --experimental_lightweight_conversion=True with constants of 30 MB (30,000,000 bytes) or more passed by reference; the export peaked at a 73.6 GB memory footprint on a 128 GB Mac. One command rebuilds the file's six sections byte for byte: nemotron_bolt_work/convert_bolt_e4b.sh in hf-to-litertlm (REPRODUCE.md).
• Speed choices (int4 file): on the Mac GPU the 2,048-token cache decoded 1.23× faster than a 4,096-token cache, and the GPU composites a further 1.31×; on the Galaxy S26 GPU the composites decoded 1.33× faster. The 512-token signature cost 0.23 to 0.26 GB of GPU memory and 0.5 s of engine creation on the Galaxy S26; on the Mac CPU without a compile cache (--cache no) the peak memory footprint was 7.76 GB, against 6.44 GB for the earlier build without the 512-token signature (means of two processes). Blocks of 128 weights instead of 32 scored 87 on GSM8K and decoded 8 % slower on the Mac GPU; int8 weights (8.21 GB) scored 93.
• NPU file: the NPU export of litert-torch (0d3869c6) with a split KV cache; the fully connected layers quantized with static int16 activations from 13 calibration prompts (three of them 700 to 2,016 tokens long); the attention mask added by broadcasting over the query heads in the 128-token prefill graph and concatenated once per head in the decode graph (the broadcast form in the decode graph stops the Qualcomm compiler plugin of LiteRT with a floating-point exception); compiled ahead of time for the SM8850 with QAIRT 2.47.0; after compilation, two cache inputs that the prefill graph lists but never reads (layer 23's keys and values) were renamed, so that the runtime creates those cache buffers from the decode graph, where they are read. The bit widths came from two failures: int4 weights on every fully connected layer except the attention projections (the NPU path quantizes per channel, not in blocks) answered 13 of the first 27 GSM8K questions on the phone, against 27 for the reference; int8 weights everywhere needed 4.48 GiB in the decode graph, above the 3.5 GiB the compiler allows for one graph, and split into two graphs the 4.7 GB context could not be created on the phone. The gate, up and vocabulary projections at int4 put the decode graph's weights at 3.21 GB.
• Tokenizer and template: the tokenizer section is the source's tokenizer.json byte for byte. The chat template is NVIDIA's with its 27 .get('…') lookups written as ['…'], which LiteRT-LM 0.18 renders; on four conversations the runtime's render equals transformers' render byte for byte. A system message renders with one space after its text (the template trims each text part and adds a space), as transformers does when the content is given as text parts.
• Logits: the decode graph of the int4 file returns the lm_head output as the logits, without Gemma 4's final logit soft cap (30 * tanh(x / 30); final_logit_softcapping: 30.0 in the source's config.json), which transformers applies after lm_head. Google's gemma-4-E4B-it.litertlm (litert-community/gemma-4-E4B-it-litert-lm) has it (a TANH between lm_head and its logits output); the int4 file has none. litert-torch's Gemma 4 decode modules call lm_head without the cap (litert-torch#1382; its fix PR, #1388, was not merged on 2026-10-27): the default one at 0a81bf6f, which made the int4 file, and the split-cache one at 0d3869c6, which made the NPU file; the NPU file's decode graph is compiled, so it is judged from that code, not read. The cap does not change which logit is largest, so greedy decoding returns the same tokens with or without it, and the greedy runs of the GSM8K table above do not depend on it. On CPU and GPU, LiteRT-LM v0.18.0 samples with top-k 1 when neither the file nor the app sets a sampler, and neither file sets one; what the NPU path samples by default was not checked. An app that sets top-k above 1 samples from the uncapped logits; #1382 shows how the cap changes the token probabilities of google/gemma-4-E4B-it in transformers. The effect on sampled text was not measured on these files.

Known runtime issues

With thinking off, from the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because NVIDIA-Nemotron-3.5-Bolt-E4B_int4.litertlm holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). With thinking on, after an answer that carries its thought (<|channel>thought … <channel|>), the runtime rebuilds the cache from the re-rendered history at the next turn, so the second turn is free of it when the conversation opens with a user message; the same source shows a zero embedding at another token from the third turn on (not measured). The NPU file is not affected: LiteRT-LM's NPU executor looks each token up in the embedding table before the token goes into the cache (read in the source of 0.18.1 and of main at 3dbb23e1). One question per conversation avoids both, as does a new conversation that starts with the earlier messages and the new question.

On LiteRT-LM releases before 0.17.0 (read in the source of 0.10.1 to 0.16.1), an answer whose last punctuation mark is a token of its own, such as a . or ? after a word, arrives without that mark: these files carry stop sequences of <turn|> and a newline with each of 13 characters in front (12 punctuation marks and a space), so the runtime holds the mark back to see whether <turn|> and a newline follow, the answer ends on <turn|> alone, and the held mark is never written. LiteRT-LM 0.17.0 and later write the held mark (StopTokenStreamFilter in runtime/core/tasks.cc, read in the source of 0.17.0 to 0.18.1). Not measured on these files: this follows from the runtime's source read with these files' stop list and tokenizer; the drop itself was seen with LiteRT-LM 0.16.1 on a Galaxy S26, on another bundle whose stop sequences have the same form.

License and changes

The model is released by NVIDIA under the OpenMDW License Agreement, version 1.1 (OpenMDW-1.1), which grants permission "to deal in the Model Materials without restriction" and asks that a distribution keep a copy of the agreement and the notices of origin. This repository carries the agreement as LICENSE and the origin and the changes as NOTICE. NVIDIA's card adds that the model "has been post-trained on the Google Gemma 4 model, and use of the Google Gemma 4 model is governed by Apache License, Version 2.0". Changes from the original work: weights converted from safetensors BF16 to LiteRT flatbuffers and quantized as described above, the image and audio encoders left out, the chat template edited as described above; the NPU file is compiled for one chip.