turintech/Nemotron-3.5-Bolt-E4B-Artemis-GGUF

TurinTech models, built with Artemis

NVIDIA's Nemotron 3.5 Bolt E4B for chat, tool calls and maths on a small GPU or a CPU, in files the size of Unsloth's.

• 4.22 GB download for the default file, Nemotron-3.5-Bolt-E4B-Artemis-Q3_K_XL.gguf.
• 3.2 GiB of GPU memory and 4.3 GiB of host memory with all layers on an NVIDIA RTX 3060 12 GB, 8,192-token context.
• Held-out KL from the original (wiki, chat, code; RTX 3060): .0574, .0414, .0376, against .0752, .0620, .0504 for Unsloth's same-size UD-Q3_K_XL. CPU task suite, on two texts that also chose the files: .0661 / .0416 against .0816 / .0609.

Run it

A chat model for llama.cpp's server with the chat template stored in the file, which is NVIDIA's template byte for byte; tested with llama.cpp b11869. Every result on this card was measured with thinking off, so the command turns it off. The files hold the text model only.

llama-server -hf turintech/Nemotron-3.5-Bolt-E4B-Artemis-GGUF:Q3_K_XL --port 8080 -c 8192 --jinja --reasoning off

Request example, thinking, other files, NVIDIA's DFlash drafter, Ollama and LM Studio

llama-server -m Nemotron-3.5-Bolt-E4B-Artemis-Q3_K_XL.gguf --port 8080 -c 8192 --jinja --reasoning off

curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "In two sentences, why is the sky blue?"}],
  "max_tokens": 256
}'

-c 8192 sets an 8,192-token context; --jinja uses the chat template stored in the file. Without --reasoning off, llama-server b11869 turns thinking on for this template (its --reasoning setting defaults to auto), and replies then start with a reasoning block and need a much larger max_tokens. Thinking on was not measured: every number on this card has thinking off, and there is no thinking-on run.

# another file, here the larger Q4_K_XL: download it, then the same command
hf download turintech/Nemotron-3.5-Bolt-E4B-Artemis-GGUF Nemotron-3.5-Bolt-E4B-Artemis-Q4_K_XL.gguf --local-dir .
llama-server -m Nemotron-3.5-Bolt-E4B-Artemis-Q4_K_XL.gguf --port 8080 -c 8192 --jinja --reasoning off

Questions, problems or a model to consider for a smaller version? Reply in our feedback thread.

Faster generation with NVIDIA's DFlash drafter

NVIDIA publishes a DFlash drafter for this model, Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf (737 MB) in nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash (revision 0949a2e, OpenMDW-1.1). It is not in this repository. It proposes the next tokens for the main file to check and does not run on its own.

hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf --revision 0949a2eb7152ec6fda1a91ff9302aecbcd464fbf --local-dir .
llama-server -m Nemotron-3.5-Bolt-E4B-Artemis-Q3_K_XL.gguf --spec-type draft-dflash --spec-draft-model Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf --spec-draft-ngl all -ngl 99 --port 8080 -c 8192 --jinja --reasoning off

On an RTX 3060 it raised generation with the Q3_K_XL file from 80.0 to 114.1 tokens per second on our 30 prompts, and used about 1.8 GiB more GPU memory (see Speed and memory on an RTX 3060). Greedy replies are not always word for word the same with it on: 11 of the 30 matched the replies without it.

Run it with Ollama

ollama run turintech/nemotron-3.5-bolt-e4b-artemis:q3_k_xl --think=false

The tags q3_k_xl and latest hold the default file, and q4_k_xl and iq3_xxs the files of those labels, each unchanged from this repository, with an 8,192-token context. Every number on this card was measured with thinking off, and Ollama turns thinking on for this model unless told otherwise: keep --think=false, or send "think": false to /api/chat ("reasoning_effort": "none" on /v1/chat/completions). Ollama formats the prompt with its own Gemma 4 renderer, not the template stored in the file. With Ollama 0.40.13 on a CPU, each file answered a short chat and a get_weather tool call with thinking off; this checks that the files work there, not their quality.

Run it with LM Studio

lms get turintech/nemotron-3.5-bolt-e4b-artemis@q3_k_xl --gguf
lms load turintech/nemotron-3.5-bolt-e4b-artemis

The LM Studio Hub entry turintech/nemotron-3.5-bolt-e4b-artemis loads the files of this repository unchanged, with an 8,192-token context and thinking off, the setting every number on this card was measured with. Keep the label in lms get (@q3_k_xl, @q4_k_xl or @iq3_xxs): without one LM Studio can pick a different file. lms load takes no label: with more than one of these files downloaded it can load another one (with all three, it loaded Q4_K_XL in a local test), so keep only the file to run, and check selectedVariant in lms ps --json after loading. A file imported straight from Hugging Face skips these settings and thinks before it answers; set the context to 8,192 and turn thinking off before chatting. With LM Studio's headless llmster 0.0.27-1 and its CPU llama.cpp engine 2.66.0 on Linux, each file, loaded through a local copy of this entry with only that file downloaded, answered a short chat and a get_weather tool call without thinking; this checks that the files work there, not their quality.

Choose this model

• Best for chat and tool calling with thinking off on a small GPU or a CPU, at the size of Unsloth's UD-Q3_K_XL. Thinking on was not measured for any file.
• Consider another file when closeness to the original matters more than size: our Q4_K_XL (5.06 GB) is the closest of our files and, with a CUDA build, slightly closer than Unsloth's UD-Q4_K_XL on held-out texts; with llama.cpp's CPU build UD-Q4_K_XL is closer by KL.
• Consider another file when speed matters more than closeness: on an RTX 3060 Unsloth's UD-Q3_K_XL generated 85.3 against 80.0 tokens per second for our Q3_K_XL, and processed 512-token prompts at 3,298 against 2,904.
• Consider another file when memory matters more than closeness: UD-Q3_K_XL used 2,956 against 3,314 MiB of GPU memory on the RTX 3060, and peaked at 4.3 against 4.9 GiB on the CPU with --cache-ram 0.
• Consider another file when size matters most: IQ3_XXS (3.70 GB) is the smallest.

Files

Every file we publish for this model. Sizes are download sizes, not memory.

• With llama.cpp or a Hugging Face pull, add the file's quant label as the tag: -hf turintech/Nemotron-3.5-Bolt-E4B-Artemis-GGUF:Q3_K_XL.
• The Ollama tag runs our package, which holds the same file: ollama run turintech/nemotron-3.5-bolt-e4b-artemis:<tag>.
• LM Studio's Hub entry turintech/nemotron-3.5-bolt-e4b-artemis offers every model file above, without the importance matrix.

Without a tag, -hf and hf.co fetch Nemotron-3.5-Bolt-E4B-Artemis-IQ3_XXS.gguf, not the default file, so always add the tag.

The Q3_K_XL and Q4_K_XL importance matrices are the same file. NVIDIA's DFlash drafter is in NVIDIA's own repository (see Run it).

How the files compare

"Saved" is the file-size reduction against the BF16 original (our BF16 GGUF of the text model, 15.05 GB). Variant links download the exact GGUF; full filenames are in the commands and checksums. Each label is the label of the Unsloth file of the same size that the file is compared with, explained under "File names, sizes and SHA-256 checksums" at the end of the card.

"Same next token" is how often a file predicts the same next token as the original, and KL measures how far its next-token probabilities are from the original's (lower is closer), both on the task suite's two texts below, measured on a CPU. These texts were also used to choose the files, on an RTX 3060, where all three files were closer to the original than here; KL on held-out texts measures the files on three texts that were not used.

Results

Our default file against Unsloth's file of the same size, NVIDIA's own Q4_K_M and the original. KL: held-out chat text, RTX 3060. Tasks: CPU task suite on two Ryzen 9 5900X machines, where the first 200 tool calls and GSM8K are selection data.

No significant difference on any task: no paired test separated our Q3_K_XL from these files. How it was tested.

Next-token agreement with the BF16 original on the OASST1 chat text, 60 chunks of 512 tokens, in the CPU task suite (AMD Ryzen 9 5900X, llama.cpp b11869). The text was also used to choose our files, on an RTX 3060, so this is not an independent test, and token agreement is not task accuracy: after correction for the 188 paired tests, no task result separated our files from the original or any public file. On this CPU our default Q3_K_XL and our IQ3_XXS agree more often than Unsloth's files of their size; our Q4_K_XL agrees slightly less often than Unsloth's UD-Q4_K_XL, the reverse of the CUDA build on the RTX 3060, for a reason not established. The BF16 original (15.05 GB, the reference) and bartowski's IQ3_M and Q3_K_M are left out of the plot; all values are in How it was tested.

Evaluation details and limitations

How it was tested

Recommendation notes

Download Nemotron-3.5-Bolt-E4B-Artemis-Q3_K_XL.gguf (4.22 GB) and run it with llama.cpp's chat server, as in Run it. On the held-out texts, on an RTX 3060, its KL from the original was .0574, .0414 and .0376 (wiki, chat, code), 24%, 33% and 26% below Unsloth's UD-Q3_K_XL of the same size (.0752, .0620 and .0504). In the CPU task suite below it was .0661 / .0416 (wiki / chat) against .0816 / .0609, on the two texts that also chose the files. On tasks there was no significant difference: it got 372 of 400 tool calls, 228 of 250 GSM8K problems, 111 of 280 MMLU-Pro questions and 466 of 541 IFEval prompts right, against 372, 228, 110 and 468 for UD-Q3_K_XL, 369, 231, 108 and 471 for NVIDIA's Q4_K_M and 370, 228, 113 and 469 for the original, and no paired test separated it from any of them. The first 200 tool calls and the GSM8K problems are selection data.

Q4_K_XL (5.06 GB) got 371, 232, 111 and 467 right, and no paired test separated it from Q3_K_XL. With a CUDA build it is the closest of our files to the original and slightly closer than Unsloth's UD-Q4_K_XL on the held-out texts (on wiki, within the measurement's ±). With llama.cpp's CPU build, UD-Q4_K_XL is closer by KL (.0307 / .0206 against .0375 / .0241). Tasks show no significant difference between the two. The cause is not established, and nothing is claimed for Apple Metal or other backends. IQ3_XXS (3.70 GB) is the smallest: it got 373, 231, 115 and 470 right and is further from the original (.1039 / .0716 on the CPU); no paired test separated it from the original.

Use the exact filename or the :Q3_K_XL tag in the commands above; without a tag, an automatic download selects the IQ3_XXS file.

Every number on this card was measured with thinking off (--reasoning off, which sets the template's enable_thinking to false). There is no thinking-on run, so answers, accuracy and output length with thinking on are not established for these files.

How the files were chosen

Each file was built to the size of one public file of this model, Unsloth's UD-IQ3_XXS, UD-Q3_K_XL and UD-Q4_K_XL (unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF revision b738967), and chosen by its KL divergence from the BF16 original on the two texts of the task suite below, on an NVIDIA RTX 3060 with llama.cpp b11869 built with CUDA. The first 200 BFCL tool-call tasks and the 250 GSM8K problems of the suite were guardrails in that choice: a candidate had to get at least 199 GSM8K problems right, and tool calls below 188 of 200 counted against it. On that panel each of our files had lower KL than the Unsloth file of its size on both texts: Q4_K_XL .0282 / .0182 against .0294 / .0204, Q3_K_XL .0558 / .0372 against .0765 / .0585, and IQ3_XXS .0940 / .0675 against .1952 / .1552 (wiki / chat). The task suite measures the same texts again on a CPU. The published files hold the same tensors as the files scored there; only the general.name header field differs.

So the KL and same-next-token columns, BFCL tasks 0 to 199 and GSM8K are selection data for our files. BFCL tasks 200 to 399, MMLU-Pro, IFEval and the held-out KL texts below were not used in the choice.

KL on held-out texts

Three texts that no step of the choice read were declared, with the files to score and the pass rule, at 16:59 UTC on 28 October 2026, before any file was scored on them:

• held-out wiki: the WikiText-2 test set after the part that the selection and the task suite score;
• held-out chat: the OASST1 validation conversations after the part that the selection and the task suite score;
• code: Python from psf/requests v2.34.2 (half of the tokens), JavaScript from expressjs/express v5.3.0 and Go from spf13/cobra v1.10.2 (a quarter each).

A check written before the run found no line of 16 or more characters and no 64-character passage of these texts in the calibration text of any of our files (the CPython source included) or in the selection texts; lines and conversations that shared text were removed while the texts were built. Each file was scored against the BF16 original on 60 chunks of 512 tokens per text with llama-perplexity, on an NVIDIA RTX 3060 with the same llama.cpp b11869 CUDA build as the choice. ± is llama-perplexity's uncertainty of each mean; sizes are in the task suite table below.

The rule was that each of our files has a lower mean KL than the Unsloth file of its size on each text, and all nine comparisons pass. The default Q3_K_XL's KL is 24%, 33% and 26% lower than UD-Q3_K_XL's on wiki, chat and code. Q4_K_XL's margin over UD-Q4_K_XL on the held-out wiki text, .0008, is within the measurement's ± (.0011 for the two means combined), so that comparison alone does not separate the two files; on chat and code its KL is 14% and 8% lower. Same next token follows the KL order except for Q4_K_XL on chat, 95.06% against 95.11%, within the ±. NVIDIA's Q4_K_M was scored without a claim. Unsloth's calibration text is not public; if it held any of these texts, that would favour the Unsloth files.

Task suite on two AMD Ryzen 9 5900X machines

Every file, ours and public, was compared with the same BF16 reference between 26 and 29 October 2026 on two identical AMD Ryzen 9 5900X machines (12 cores, 31 GiB RAM, Ubuntu 22.04), CPU only, 12 threads, with the same llama.cpp b11869 CPU build (d327bf1f0). The second machine scored bartowski's IQ4_XS and Q3_K_M and ggml-org's Q8_0; the first scored the other twelve rows, ours included. A cross-check matched: one file run on both machines (5 KL chunks per text and the first 25 items of each task) gave identical KL lines and replies, and its only different verdict was the unseeded IFEval letter explained below. The KL runs used llama-perplexity on 512-token chunks with flash attention on and an F16 KV cache. The task runs used llama-server with context 8192, flash attention on, F16 KV cache, greedy decoding (temperature 0, top_k 1, seed 20261020), thinking off (--reasoning off) and one slot, with NVIDIA's chat template, which is the same template as the one stored in our files. Replies were capped at 1,024 tokens for tool calls, 768 for GSM8K and 2,048 for IFEval.

• KL divergence and next-token agreement over 60 chunks of 512 tokens from each of two human-written texts: the WikiText-2 test set (Wikipedia) and OpenAssistant OASST1 validation conversations (English). These are the same two texts, byte for byte, as on our Gemma 4 E4B card.
• Native tool calling on BFCL simple (Python), all 400 tasks, through llama-server's OpenAI-style tools API, scored with BFCL's own checker (bfcl_eval 2026.3.23). A reply with no tool call counts as wrong.
• GSM8K maths, the first 250 test problems.
• MMLU-Pro, 280 questions (20 from each category), zero-shot with a direct letter answer and replies capped at 8 tokens. This is not the leaderboard protocol. A reply without a letter counts as wrong, and even the original gave no letter on 59 of 280 (21 to 81 across the other files), so these counts mix answer format with knowledge.
• IFEval, all 541 prompts, strict prompt-level accuracy with the lm-evaluation-harness v0.4.13 checks.

IFEval verdicts are rescored for two items. For a letter-frequency instruction whose letter is not a to z (# in the prompt with key 1122, ! in key 1129), the checker picks a random letter, and nothing seeded that choice, so the verdict could change from one run to the next. Every row here is rescored with the checker's random letter seeded from the item id (b and u), the same for every row; the other 539 items keep their recorded verdicts. The checker's language checks (95 items: 39 lowercase, 25 capital-letter and 31 response-language checks) use langdetect, which has its own random generator that is not seeded, so a few of those verdicts can vary between runs. They are kept as recorded; one such flip was seen between two runs of the same reply. The rescoring moved three rows by one prompt each: the original from 470 to 469, bartowski's Q4_K_M from 464 to 463 and our Q3_K_XL from 467 to 466. Every IFEval count and paired test below uses the rescored verdicts.

Peak memory comes from a separate llama-server run per file with the prompt cache off (--cache-ram 0), answering the first 25 items of each task. It is the server's peak resident memory, which counts the parts of the memory-mapped model file that were read.

Selection data for our files: the KL and same-next-token columns, tool-call tasks 0 to 199 and GSM8K. Not used in the choice: tool-call tasks 200 to 399, MMLU-Pro, IFEval and the held-out KL texts.

With this CPU build our Q4_K_XL is further from the original than Unsloth's UD-Q4_K_XL on both texts (.0375 / .0241 against .0307 / .0206), the reverse of the CUDA build on the RTX 3060 in How the files were chosen and KL on held-out texts; no paired task test separated the two. On the same two texts, our three files' KL is 6% to 33% higher here than on the RTX 3060, against 1% to 7% for Unsloth's three; the cause has not been established, and nothing is claimed for Apple Metal or other backends. Our Q3_K_XL and IQ3_XXS stay below the Unsloth files of their size here too.

Each cell gives the items only the first file got right, the items only the second got right, and p. The suite report made 188 paired tests (each of our three files against the original and every public file, and each public file against the original, on four tasks); after a Bonferroni correction for 188 (p < 0.00027), no difference remains. The last row compares two of our files; the report tests our files against each other separately from the 188. Before the correction 4 had p < 0.05, none of them in the table above: Unsloth's UD-IQ3_XXS against the original on tool calls (12 / 2, p=0.013), our Q4_K_XL and Q3_K_XL against Unsloth's UD-IQ3_XXS on tool calls (3 / 12, p=0.035 and 2 / 10, p=0.039), and our IQ3_XXS against Unsloth's UD-IQ2_M on MMLU-Pro (17 / 6, p=0.035). Unsloth's UD-IQ3_XXS got the most tool calls right, 380 of 400.

Public files, each checked by SHA-256: nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB revision 1db54d5 (Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf), unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF revision b738967 (NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q4_K_XL.gguf, -UD-Q3_K_XL.gguf, -UD-IQ3_XXS.gguf, -UD-IQ2_M.gguf), bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF revision 7a243bc (NVIDIA-Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf, -IQ4_XS.gguf, -IQ3_M.gguf, -Q3_K_M.gguf, -IQ3_XXS.gguf) and ggml-org/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF revision e247355 (NVIDIA-Nemotron-3.5-Bolt-E4B-Q8_0.gguf). This suite measured no speeds; speeds are in the next section.

Speed and memory on an RTX 3060

One NVIDIA GeForce RTX 3060 12 GB (driver 595.71.05, 170 W power limit) in an AMD Ryzen 9 5900X desktop with 31 GiB of RAM, llama.cpp b11869 (d327bf1f0) built with CUDA 12.2, every layer on the GPU, measured on 28 October 2026 with nothing else running. This measures speed and memory only, not quality; compare rows within this table.

• llama-bench: -p 512 -n 128 -r 5 -ngl 99 -t 12 -fa auto, prompt processing of 512 tokens and generation of 128, five repetitions. Prefill shows the mean and standard deviation; the generation standard deviations were 0.2 tokens per second or less.
• llama-server: -c 8192 -np 1 -ngl 99 -t 12 --jinja --reasoning off, with NVIDIA's chat template, on 30 prompts: the first 15 GSM8K test problems and the first 15 IFEval prompts, one user turn each, greedy (temperature 0, top_k 1, seed 20261020), prompt cache off, at most 256 new tokens, one request at a time. About 18 of the 30 replies stop at 256 tokens. Generation speed is the server's generated tokens over its generation time, summed over the 30 prompts.
• With DFlash: the same server command plus --spec-type draft-dflash --spec-draft-model Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf --spec-draft-ngl all, every other speculative setting at llama.cpp's default (up to 3 drafted tokens, greedy). Acceptance is the share of drafted tokens the main file kept.
• Memory: the server process's peak GPU memory from nvidia-smi, sampled every 0.5 s, and its peak resident host memory after the last request, which counts the memory-mapped model file.

File sizes are in the task suite table above.

The drafter raised generation speed by 28% to 56% across these files and used about 1,800 MiB more GPU memory. A longer draft (--spec-draft-n-max 7, the drafter's block size minus one, the most llama.cpp allows) lowered acceptance to 32 to 33% and was slower than the default for every file (92.1 to 101.0 tokens per second).

With the drafter, 6 to 12 of the 30 greedy replies (11 for our default Q3_K_XL) were byte for byte the same as without it. Checking several drafted tokens at once takes different GPU kernel paths than generating one token at a time, so nearly tied tokens can come out differently. This table does not measure quality with the drafter.

GPU memory stays below the file size for the larger files because llama.cpp keeps the input embedding tables, including this model's large per-layer embedding table, in host memory; peak host memory is close to the file size. The 30-prompt prefill rates (about 1,000 tokens per second) come from prompts of about 70 tokens; llama-bench's prefill 512 is the throughput figure.

Public files: as in the task suite, plus NVIDIA's drafter (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash revision 0949a2e, Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf, 736,812,096 bytes), each checked by SHA-256 on the measuring machine.

Calibration

The importance matrices were computed from WikiText-2 raw train (Salesforce/wikitext, wikitext-2-raw-v1, CC BY-SA 3.0 and GFDL) and human-written OpenAssistant OASST1 train conversations (Apache-2.0; synthetic and deleted messages excluded), and for the IQ3_XXS file also from the CPython 3.11 standard library source (Python Software Foundation License). Each file's importance matrix is published beside it as Nemotron-3.5-Bolt-E4B-Artemis-<LABEL>-imatrix.gguf_file; the Q3_K_XL and Q4_K_XL matrices are the same file. A 13-gram check found no overlap between the calibration text and the suite's chat text, all 400 BFCL tasks, GSM8K, MMLU-Pro or IFEval, and two shared 13-grams with the WikiText-2 test text. The GGUF headers hold no local paths; the imatrix.datasets field of each matrix names the public datasets above.

Closeness to the original

The closeness numbers from the tables above, in one fixed layout with the protocol behind each table.

Protocols and standard tables

• Mean KLD: the mean KL divergence between the reference's and the file's next-token distributions, computed and averaged as each protocol's Tool line says. Lower is closer.
• 99th-percentile KLD: the 99th percentile of the per-token values. Lower is closer.
• Same top p: llama.cpp's name for top-1 agreement, the share of scored tokens where the file and the reference pick the same most likely token. Higher is closer.
• PPL and PPL ratio: the file's perplexity on the text, and that perplexity divided by the reference's (1 means the same). They measure prediction of the text, not closeness.

n/a means the card's tables or our run records do not have that value.

Protocol rtx3060-heldout-2026-10-28

NVIDIA RTX 3060, three held-out texts, 28 October 2026.

• Reference: BF16 GGUF conversion of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 revision 49843eed8db1f1c616c47af75cfd36e07aec0667 (text only, 720 tensors, 15,053,100,000 bytes), converted with llama.cpp b11869, SHA-256 ef1f3436bb992e0b19ab82d002446f1b87af008811e8a96f64ed955bb28aa4d6
• Text wiki: WikiText-2 test after the part that the selection and the task suite score; dataset: Salesforce/wikitext at b08601e04326c79dfdd32d625aee71d232d685c3; config: wikitext-2-raw-v1; split: test, from the first article after the scored part; chunks: 60; context (tokens): 512; text SHA-256: 20b1259fcda71ded703f17a8db065804102e0b604b8ddf0fbfb789ae80f176e5
• Text chat: OpenAssistant validation conversations after the part that the selection and the task suite score (English threads as User/Assistant text); dataset: OpenAssistant/oasst1 at fdf72ae0827c1cda404aff25b6603abec9e3399b; split: validation, 89 threads after the scored part (6 that share text with OASST1 train removed); chunks: 60; context (tokens): 512; text SHA-256: 8e1d300df46318e47c1c961b931762fcd6023dea6e2d45b15c0182e734bf5425
• Text code: Python from psf/requests (half of the tokens), JavaScript from expressjs/express and Go from spf13/cobra (a quarter each); dataset: GitHub source: psf/requests v2.34.2 (commit 6e83187b), expressjs/express v5.3.0 (commit cf6722bf), spf13/cobra v1.10.2 (commit 88b30ab8); config: whole files in path order, each opened by a comment line with its path, licence comment blocks removed; split: requests/src/requests/.py, then express index.js and lib/**/.js, then cobra's top-level *.go files without tests; chunks: 60; context (tokens): 512; text SHA-256: 1c6b254f1a096d791731064ef67e8d1782e6cbc4ca1a107b978c1c32620df0e1
• Tool: llama-perplexity --kl-divergence against the reference's saved log-probabilities, chunks of 512 tokens; Mean KLD and Same top p are llama.cpp's own summary lines, and the card's tables also print their ±
• llama.cpp build: b11869 (commit d327bf1f0780f87b8c01b367b61853b0c291d7c4), built with CUDA 12.2
• Command: llama-perplexity -m <file.gguf> --kl-divergence-base <reference-text.kld> --kl-divergence -c 512 --chunks 60 -ngl 99 -t 16 --override-kv tokenizer.ggml.add_bos_token=bool:true; the reference logs were written by the BF16 file with llama-perplexity -m <BF16.gguf> -f <text> --kl-divergence-base <reference-text.kld> -c 512 --chunks 60 -ngl 40 -t 20 --override-kv tokenizer.ggml.add_bos_token=bool:true
• Hardware: One NVIDIA GeForce RTX 3060 12 GB in an AMD Ryzen 9 5900X workstation; every layer of each scored file on the GPU, 40 layers of the BF16 reference on the GPU and the rest on the CPU
• Settings: CUDA build, 512-token chunks, BOS added, flash attention and KV cache at llama-perplexity's defaults
• KL unit: nats (natural log)
• Measured by: TurinTech downloaded the Unsloth and NVIDIA files at the revisions below, checked each by SHA-256 and measured every file in this table with the same reference, build and protocol
• Selection: not used: no step of the choice read these texts; they were declared, with the files and the pass rule, at 16:59 UTC on 28 October 2026, before any file was scored on them
• Source on this card: the table columns "KL, held-out wiki", "Same next token, held-out wiki", "KL, held-out chat", "Same next token, held-out chat", "KL, code", "Same next token, code"
• Notes: A check written before the run found no line of 16 or more characters and no 64-character passage of these texts in the calibration text of any of our files or in the selection texts; lines and conversations that shared text were removed while the texts were built. Unsloth's calibration text is not public; if it held any of these texts, that would favour the Unsloth files. NVIDIA's Q4_K_M was scored without a claim.
• Comparison files: Unsloth: unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at b738967d6da3, NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q4_K_XL.gguf, SHA-256 84ff34f5834b929043f5ae74799e8d562d4056998f6a5923ceb44831e8ad84d6; Unsloth: unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at b738967d6da3, NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q3_K_XL.gguf, SHA-256 3f8032b9d2c18536c4ac51b97959a563cc27434de99823640c98147edc62ce9b; Unsloth: unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at b738967d6da3, NVIDIA-Nemotron-3.5-Bolt-E4B-UD-IQ3_XXS.gguf, SHA-256 e678b2bdc79a31a6f954062fedeb43b41e5bab105c3743a4d1a1926f2a9be173; NVIDIA: nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB at 1db54d5b1253, Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf, SHA-256 c8af2b8aea10a690a1d9491db744a9446660173cb38fc489612b556615a69612

Protocol rtx3060-heldout-2026-10-28, text wiki (WikiText-2 test after the part that the selection and the task suite score):

Protocol rtx3060-heldout-2026-10-28, text chat (OpenAssistant validation conversations after the part that the selection and the task suite score (English threads as User/Assistant text)):

Protocol rtx3060-heldout-2026-10-28, text code (Python from psf/requests (half of the tokens), JavaScript from expressjs/express and Go from spf13/cobra (a quarter each)):

Protocol ryzen5900x-2026-10

AMD Ryzen 9 5900X, 26 to 29 October 2026.

• Reference: BF16 GGUF conversion of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 revision 49843eed8db1f1c616c47af75cfd36e07aec0667 (text only, 720 tensors, 15,053,100,000 bytes), converted with llama.cpp b11869, SHA-256 ef1f3436bb992e0b19ab82d002446f1b87af008811e8a96f64ed955bb28aa4d6
• Text wiki: WikiText-2 test; dataset: Salesforce/wikitext at b08601e04326c79dfdd32d625aee71d232d685c3; config: wikitext-2-raw-v1; split: test; chunks: 60; context (tokens): 512; text SHA-256: bbf94c53a05abe9ee670d3b6343608095822c85e26de37c70b24fc571964574a
• Text chat: OpenAssistant validation conversations (English threads as User/Assistant text); dataset: OpenAssistant/oasst1 at fdf72ae0827c1cda404aff25b6603abec9e3399b; split: validation; chunks: 60; context (tokens): 512; text SHA-256: 0a3499a7eaead8f4a40655a868e7faee1ee664e258d09a77cd8f3d97216fc8b7
• Tool: llama-perplexity --kl-divergence against the reference's saved log-probabilities, chunks of 512 tokens; Mean KLD and Same top p are llama.cpp's own summary lines
• llama.cpp build: b11869 (commit d327bf1f0780f87b8c01b367b61853b0c291d7c4), CPU build
• Command: llama-perplexity -m <file.gguf> -f <text> --kl-divergence-base <reference-text.kld> --kl-divergence -c 512 --chunks 60 -t 12 -fa on -ctk f16 -ctv f16; the reference logs were written by the BF16 file with the same command without --kl-divergence (the suite harness, score_file.sh)
• Hardware: Two identical AMD Ryzen 9 5900X machines (12 cores, 31 GiB RAM, Ubuntu 22.04), CPU only, 12 threads, the same llama.cpp build; the second machine scored bartowski IQ4_XS and Q3_K_M and ggml-org Q8_0, the first every other file; a 5-chunk slice of one file gave identical KL lines on both
• Settings: CPU only, 12 threads, flash attention on, F16 K and V cache, 512-token chunks
• KL unit: nats (natural log)
• Measured by: TurinTech downloaded the NVIDIA, Unsloth, bartowski and ggml-org files at the revisions below, checked each by SHA-256 and measured every file in this table with the same reference, build and protocol
• Selection: both texts were used to choose our files, with KL measured on an NVIDIA RTX 3060 with a CUDA build of the same llama.cpp release
• Source on this card: the table columns "KL wiki", "KL chat", "Same next token, wiki", "Same next token, chat"
• Comparison files: NVIDIA: nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB at 1db54d5b1253, Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf, SHA-256 c8af2b8aea10a690a1d9491db744a9446660173cb38fc489612b556615a69612; ggml-org: ggml-org/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at e24735556471, NVIDIA-Nemotron-3.5-Bolt-E4B-Q8_0.gguf, SHA-256 64504effb9a71a991e487ac4e4cf827485e78bf577efb0a6edf67ce7e1ba84a8; Unsloth: unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at b738967d6da3, NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q4_K_XL.gguf, SHA-256 84ff34f5834b929043f5ae74799e8d562d4056998f6a5923ceb44831e8ad84d6; Unsloth: unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at b738967d6da3, NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q3_K_XL.gguf, SHA-256 3f8032b9d2c18536c4ac51b97959a563cc27434de99823640c98147edc62ce9b; Unsloth: unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at b738967d6da3, NVIDIA-Nemotron-3.5-Bolt-E4B-UD-IQ3_XXS.gguf, SHA-256 e678b2bdc79a31a6f954062fedeb43b41e5bab105c3743a4d1a1926f2a9be173; Unsloth: unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at b738967d6da3, NVIDIA-Nemotron-3.5-Bolt-E4B-UD-IQ2_M.gguf, SHA-256 070456a111a8431a32ba72a506f76e71f5ada7c684371fd63cf9083b24551795; bartowski: bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at 7a243bc11b80, NVIDIA-Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf, SHA-256 5f0f982d11bc1408783d13e4440b06e6c1e989024ac26f9a2999cb4c496bedcc; bartowski: bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at 7a243bc11b80, NVIDIA-Nemotron-3.5-Bolt-E4B-IQ4_XS.gguf, SHA-256 7874aa84419f61b3ced4cbe762a171ccc3d2930381bf58fd0685dde888e89d46; bartowski: bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at 7a243bc11b80, NVIDIA-Nemotron-3.5-Bolt-E4B-IQ3_M.gguf, SHA-256 052ecb87ccf74c9d2e444f97cd6bcd2b60e1844cf5f841bfa9a944633d8e4abd; bartowski: bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at 7a243bc11b80, NVIDIA-Nemotron-3.5-Bolt-E4B-Q3_K_M.gguf, SHA-256 eddcdcf60944ee9616e8e3dd348c480ea02a8775e38f2b65f03cbd8eb347344f; bartowski: bartowski/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF at 7a243bc11b80, NVIDIA-Nemotron-3.5-Bolt-E4B-IQ3_XXS.gguf, SHA-256 d2e396cc59269477f3447c1a1c07ba152f62412e8654d2c556485e9ac3a54685

Protocol ryzen5900x-2026-10, text wiki (WikiText-2 test):

Protocol ryzen5900x-2026-10, text chat (OpenAssistant validation conversations (English threads as User/Assistant text)):

Memory

On an NVIDIA RTX 3060 12 GB with llama.cpp b11869, every layer on the GPU and an 8,192-token context (-c 8192 -np 1 -ngl 99), llama-server peaked at 3.2 GiB of GPU memory and 4.3 GiB of host memory with the default Q3_K_XL file, 3.8 GiB and 5.0 GiB with Q4_K_XL, and 3.0 GiB and 3.8 GiB with IQ3_XXS, over 30 short prompts. NVIDIA's DFlash drafter adds about 1.8 GiB of GPU memory. llama.cpp keeps this model's per-layer embedding table in host memory even with every layer on the GPU, so host memory stays close to the file size. The full table is in Speed and memory on an RTX 3060.

On an AMD Ryzen 9 5900X, CPU only, with an 8,192-token context and the prompt cache off (--cache-ram 0), llama-server peaked at 4.9 GiB with the default Q3_K_XL file, 5.2 GiB with Q4_K_XL and 4.3 GiB with IQ3_XXS, against 9.2 GiB for the BF16 original, answering the first 25 items of each task. The commands on this card keep llama-server's default 8 GiB prompt cache, and memory can grow as it fills: over the full task runs with that cache, the peak was 13.2 GiB with Q3_K_XL, 13.6 GiB with Q4_K_XL and 12.6 GiB with IQ3_XXS (18.0 GiB for the original). Add --cache-ram 0 to the command to stay near the lower figures.

File names, sizes and SHA-256 checksums

File names

Each file is named after the public file it was built to match in size and is compared with: Unsloth's UD-IQ3_XXS, UD-Q3_K_XL and UD-Q4_K_XL of this model, without the UD- prefix. Each of ours is 0.008% to 0.10% smaller than its Unsloth counterpart. The labels give the size of the compared file, not the file's tensor types: Q3_K_XL and Q4_K_XL are Unsloth's names for its dynamic files, not llama.cpp quantization types, and our IQ3_XXS file holds mostly IQ4_XS, IQ2_S, Q5_K and Q4_K tensors. This is an exception to the size-class naming on our other cards.

By that size-class rule the files would be IQ2_S, Q2_K_S and IQ4_XS. The rule was measured here with stock llama-quantize files made from the same BF16 with llama.cpp b11869, because this model keeps 37.5% of its weights in a per-layer embedding table, which stock llama-quantize stores at Q5_K or Q6_K, so stock files of this model are much larger than the usual bits-per-weight figures suggest. The general.file_type field in every file reads Q4_K_M.

Bits per weight are for the whole file over the model's 7,518,069,290 weights.

File checksums

Artemis

Teams use Artemis to test and improve code against their own objectives. See Artemis.

About and licence

Nemotron 3.5 Bolt E4B is NVIDIA's small chat model, post-trained by NVIDIA from Google's Gemma 4 E4B. These files are made from NVIDIA's BF16 checkpoint (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, revision 49843ee).

This is an independent TurinTech conversion, not affiliated with or endorsed by NVIDIA or Google. The files are standard GGUF and run on stock llama.cpp. Changes from the original: the text model is converted to GGUF and quantized for llama.cpp; the vision and audio encoders of NVIDIA's checkpoint are not included. See NOTICE.

OpenMDW License Agreement 1.1, the licence NVIDIA releases this model under (OpenMDW-1.1); see LICENSE and NOTICE. Google's Gemma 4 E4B, which NVIDIA started from, is released under Apache-2.0.

When serving these files to others, for example through an API, an inference provider or OpenRouter, we'd appreciate a mention of TurinTech (turintech.ai) as the source of the quantization.

These files keep improving.