New (2026-11-20): a full quant ladder (
Q8_0down toIQ2_XS) in the upstream llama.cpp layout (PR #32588) is at drofn/SenseNova-Loop-9B-GGUF. Those files need a PR #32588 build and do not load with the fork used here, and this repo's files do not load in the PR build. On an RTX 4060 the PR build is currently slower than our 8GB-v3 file on the patched fork, because the PR has no CUDA kernels for the mHC routing yet.
All sizes and versions: collection.
Version 3 of the 8 GB SenseNova-Loop-9B quants (v2, v1). Same context as v2 (104,448 tokens), about the same speed, all layers on the GPU, and shared GPU memory stayed under the same 200 MiB limit as in v2 (see "VRAM check"). The difference is how close the output stays to Q8_0:
New (2026-11-20): an HQ file for people who want closer-to-Q8_0 output and can live with a shorter context. It runs with a q8_0 KV cache at 57,344 tokens (the largest tested context that fit), and its output is 15–28% closer to Q8_0 than v3 as each file is run (see "HQ variant").
All numbers come from one RTX 4060 8GB under Windows with no display attached, except the HQ KLD comparison (RTX 5060 Ti, see below). All are single runs.
| File | Size | Use with | Context on 8 GB (MTP on) | Generation, MTP on / off |
|---|---|---|---|---|
| SenseNova-Loop-9B-8GB-v3.gguf | 6.19 GB | text | 104,448 | 47.0 / 25.4 tok/s |
| SenseNova-Loop-9B-8GB-v3-HQ.gguf | 6.51 GB | text, KV q8_0 | 57,344 | 46.0 tok/s / not measured |
| v3 + mmproj-SenseNova-Loop-9B-F16.gguf (0.92 GB) | text + images | 57,344 | 46.9 tok/s (MTP on, 1080p image prompt) |
evals/sizes.txt). llama.cpp keeps that table in system RAM: two test builds that differed only in the embedding type used the same dedicated VRAM (7,873 MiB after the long test, evals/v3-adim2.log). After the long test, dedicated VRAM went from 7,851 MiB (v2) to 7,903 MiB (v3).1bb6afd0…1481, the same file as in v2).llama-perplexity --kl-divergence against Q8_0 logits, 40 chunks of 512 tokens per set. Lower is better.
| Set | What it is | Lines also in calibration text¹ | v2 (8GB-v2) | v3 | Change |
|---|---|---|---|---|---|
| 1 | Turkish and English Wikipedia prose (the v1/v2 "set 1") | 0 / 365 | 0.0703 | 0.0566 | −19.5% |
| 2 | English Wikipedia and tool-calling dialogues, glaive-function-calling-v2 (the v1/v2 "set 2"; the scored chunks do not reach the Python at the end of the file) | 62 / 1,343 | 0.0762 | 0.0612 | −19.8% |
| 3 | Python source from llama.cpp's converter: base.py, qwen.py, llama.py (the 6GB card's "set 3") |
0 / 1,103 | 0.0359 | 0.0282 | −21.5% |
| 4 | held out: a later part of the same code file², not used for any decision | 0 / 864 | 0.0425 | 0.0337 | −20.7% |
¹ Unique lines longer than 40 characters that also appear verbatim in the calibration text. Checked over the first 90,000 characters of set 1, all of set 2, the part of the code file before the set 4 cut (set 3), and set 4. Set 2 partly overlaps. Sets 1, 3 and 4 showed no such overlap in the checked text.
² Set 3 scores the first 20,480 tokens of the code file. Set 4 starts at token 36,952 (the file has 64,026 tokens), so the two do not overlap.
| Set | 99th-percentile KLD, v2 → v3 | Same top token, v2 → v3 |
|---|---|---|
| 1 | 0.644 → 0.494 | 88.46% → 89.75% |
| 2 | 1.604 → 1.081 | 94.48% → 94.91% |
| 3 | 0.357 → 0.278 | 93.47% → 94.18% |
| 4 | 0.457 → 0.388 | 93.34% → 93.82% |
The recipe was chosen on sets 1–3, so set 4 is the honest check. It shows the same ~20% as the others.
SenseNova-Loop-9B-8GB-v3-HQ.gguf uses the v3 recipe with two changes, and the same imatrix as v3:
| Tensors | v3 | HQ |
|---|---|---|
output (the output head, kept on the GPU) |
Q4_K | Q6_K |
loop blocks 20–27: ffn_down |
Q4_K | Q5_K |
That adds about 0.31 GB of weights on the GPU, so the KV cache has to shrink. HQ is meant to run with a q8_0 KV cache (v3 uses q4_0), which costs more VRAM per token. The largest tested context that fit was 57,344 tokens. Full arguments: recipes/8GB-v3-HQ.txt.
KLD vs Q8_0, same stand. Both files were scored on one RTX 5060 Ti 16GB on the same sets as above. The 512-token rows use the same Q8_0 reference logits as above; the 4,096-token rows use reference logits from the same Q8_0 at that length (evals/v3-HQ-kld.log). The v3 numbers on the 5060 Ti differ slightly from the 4060 table above (for example 0.0569 vs 0.0566 on set 1).
| Setting | Set 1 | Set 2 | Set 3 | Set 4 |
|---|---|---|---|---|
| 512-token chunks, each file with its own KV cache (v3 q4_0, HQ q8_0) | 0.0637 → 0.0502 (−21%) | 0.0748 → 0.0635 (−15%) | 0.0347 → 0.0251 (−28%) | 0.0390 → 0.0301 (−23%) |
| 4,096-token chunks, own KV cache (10 chunks; 6 for set 4) | 0.0891 → 0.0698 (−22%) | 0.0694 → 0.0519 (−25%) | 0.0925 → 0.0785 (−15%) | 0.0314 → 0.0241 (−23%) |
| 512-token chunks, F16 KV cache for both (weights only) | 0.0569 → 0.0504 (−12%) | 0.0639 → 0.0615 (−4%) | 0.0286 → 0.0257 (−10%) | 0.0333 → 0.0297 (−11%) |
The 99th-percentile KLD and the same-top-token rate moved the same way on every row (evals/v3-HQ-kld.log). Most of the gain as run (52–79% of it at 512 tokens) comes from the KV cache: q4_0 raises v3's KLD by 12–21%, while q8_0 moves HQ's by at most 3.3%. Set 4 was also looked at when HQ was chosen, so unlike in the v3 table it is not held out here. At 4,096 tokens, sets 3 and 4 overlap by about 4,000 tokens.
AtomicChat's AD-Q4_K_M was not measured against HQ. On weights alone (F16 KV), HQ is 4–12% closer to Q8_0 than v3 on the 5060 Ti. In the 4060 test below, AD-Q4_K_M was 0–7% closer than v3, on SenseNova's fork. Those are two different stands, so they do not rank HQ against AD-Q4_K_M. HQ is also the larger file on disk (6.51 vs 5.96 GB).
On the 8 GB card (RTX 4060, llama-server, MTP on, the same tests as v3; evals/v3-HQ-server.log):
| v3 (104,448, KV q4_0) | HQ (57,344, KV q8_0) | |
|---|---|---|
| Generation, 400 tokens, MTP on | 45.5 tok/s¹ | 46.0 tok/s |
| MTP drafts accepted, five prompt types | 56.2% | 51.8% |
| Needle near the limit | 3/3 at 102,903 tokens | 3/3 at 55,773 tokens |
| VRAM after the needle test, dedicated / shared | 7,905 / 180 MiB | 7,877 / 160 MiB |
¹ Re-measured on 2026-11-19 with the same scripts, together with earlier HQ candidates (last line of evals/v3-HQ-server.log); the 47.0 tok/s in "Speed" is from the original v3 test.
HQ's overall MTP acceptance rate was lower (51.8% vs 56.2%; the MTP block is the same Q4_K in both). Per prompt type it was higher on four of five and slightly lower on the Turkish prompt, where HQ drafted far more tokens. Generation speed was about the same. At 61,440 tokens, shared memory was already 288 MiB after the short tests, over this card's 200 MiB limit, so 57,344 is the published context.
Run it like v3, with q8_0 for the KV cache and a smaller -c:
./build/bin/llama-server -m ./models/SenseNova-Loop-9B-8GB-v3-HQ.gguf -ngl 99 -fa on \
-ctk q8_0 -ctv q8_0 -c 57344 -ub 128 -np 1 --jinja --reasoning-format deepseek --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 2 -ctkd q8_0 -ctvd q8_0
HQ came out of follow-up tests after v3 with other imatrices and recipes. Only the measurements of HQ and v3 are published here. All single runs.
Correction (2026-11-19). The first version of this card compared v3 with other quants using the numbers on each publisher's card and suggested v3 was on a par with AtomicChat's files. That was not checked, and the same-stand test below does not support it. AtomicChat's AD-Q4_K_M is at least as close to Q8_0 as v3, and the file is 0.23 GB smaller.
How it was measured: same four sets, same 40 × 512-token chunks, same Q8_0 reference logits as every other number on this card, same RTX 4060. AtomicChat's files use SenseNova's official tensor layout (hc_* names). The patched fork used for v3 cannot load that layout, so they were run on SenseNova's fork, blueswhen/llama.cpp @ 24ea7bb7. That is the build AtomicChat's card names. To check that the two runtimes agree, SenseNova's official Q8_0 was scored on that fork against this card's Q8_0 reference. It scored 0.0027 / 0.0090 / 0.0015 / 0.0020. That floor (it mixes the runtime difference with the difference between two separately converted Q8_0 files) is inside AtomicChat's numbers and about as large as the gaps between v3 and AD-Q4_K_M below, larger on set 2. So those gaps are within this test's resolution: read v3 vs AD-Q4_K_M as a tie or a small edge for AD-Q4_K_M, not as a precise ranking. Logs: evals/v3-atomic.log, evals/v3-atomic.csv.
| File | Size | Mean KLD (set 1 / 2 / 3 / 4) | 99th-pct KLD (set 1 / 2 / 3 / 4) | Same top token % (set 1 / 2 / 3 / 4) |
|---|---|---|---|---|
| 8GB-v3 (this repo) | 6.19 GB³ | 0.0566 / 0.0612 / 0.0282 / 0.0337 | 0.494 / 1.081 / 0.278 / 0.388 | 89.7 / 94.9 / 94.2 / 93.8 |
| AtomicChat AD-Q4_K_M | 5.96 GB | 0.0567 / 0.0594 / 0.0278 / 0.0312 | 0.554 / 1.071 / 0.313 / 0.366 | 89.9 / 95.5 / 94.9 / 94.6 |
| AtomicChat AD-IQ4_XS | 5.81 GB | 0.0658 / 0.0702 / 0.0315 / 0.0398 | 0.665 / 1.409 / 0.338 / 0.420 | 89.1 / 95.0 / 94.4 / 93.7 |
| SenseNova Q8_0 (runtime check) | 9.80 GB | 0.0027 / 0.0090 / 0.0015 / 0.0020 | 0.022 / 0.074 / 0.012 / 0.015 | 97.9 / 98.7 / 98.9 / 98.7 |
³ This includes the Q8_0 token embedding (about 1.08 GB; 0.38 GB more than v2's Q5_K one), which llama.cpp keeps in system RAM.
What this shows:
AD-Q4_K_M is at least as faithful, in a smaller file. It ties v3 on set 1 and has 3%, 2% and 7% lower mean KLD on sets 2, 3 and 4 (set 4 is the held-out set), with a 0.2–0.8 point higher same-top-token rate. These gaps are about the size of the runtime floor above, single runs. v3 has the lower 99th-percentile KLD on sets 1 and 3. Both runtimes keep the token embedding in system RAM; the GPU-resident sizes of the two files were not compared.
v3 has 10–16% lower mean KLD than AD-IQ4_XS on all four sets (AD-IQ4_XS has a slightly higher same-top-token rate on sets 2 and 3). It is also 0.38 GB larger.
On the 8 GB card at 104,448 tokens with MTP, only the v3 run stayed in VRAM. AtomicChat's files were started on SenseNova's fork with the same flags as v3 (-c 104448 -ctk q4_0 -ctv q4_0 -ub 128 -np 1 --spec-type draft-mtp --spec-draft-n-max 2 -ctkd q4_0 -ctvd q4_0):
| File, runtime | Shared GPU memory | Generation (400 tokens, MTP on) | Needle at 102,903 tokens |
|---|---|---|---|
| v3, patched fork | 180 MiB | 47.0 tok/s | 3 / 3 |
| AD-Q4_K_M, SenseNova's fork | 1,120 MiB | 6.7 tok/s | 3 / 3 |
| AD-IQ4_XS, SenseNova's fork | 978 MiB | 6.7 tok/s | 3 / 3 |
Both AtomicChat files ran and found the needles. But about 1 GB spilled into shared system memory, and generation was about 7× slower. This is a difference between the two runtimes as much as between the files. The two were not separated, and the largest context at which AtomicChat's files fit without spilling was not measured.
So: if you can use SenseNova's fork and a smaller context, AtomicChat's AD-Q4_K_M is at least as faithful as v3 and smaller. v3 is for 104K context with MTP at full speed on one 8 GB card, on the patched fork.
Other SenseNova-Loop-9B GGUFs (wepiqx's MERNIK, Harsha001, guideboardlabs, FP8Enjoyer) were not measured here. Their cards use other references and texts, so their KLD values are not comparable to this table.
Bits were moved to where the measurements said they matter, and taken away where they did not. The result fits the same context on the same card:
| Tensors | v2 | v3 |
|---|---|---|
ssm_alpha, ssm_beta in loop and output blocks (4–27, 52–55) |
Q5_K (4–11), Q4_K (12–27, 52–55) | Q8_0 (tiny tensors; blocks 0–3 were already Q8_0) |
attn_k, attn_v in loop and output blocks |
Q5_K | Q8_0 |
token_embd |
Q5_K | Q8_0 (in system RAM, no VRAM cost) |
loop blocks 4–11: ffn_gate, ffn_up |
Q4_K | Q5_K |
loop blocks 12–19: attn_qkv, attn_gate, ssm_out |
Q4_K | Q5_K |
loop blocks 12–27: ffn_gate, ffn_up |
Q4_K | IQ4_XS |
MTP block (blk.56): the seven weight matrices and eh_proj |
Q6_K, eh_proj Q8_0 |
Q4_K (norms stay F32) |
| imatrix | v1's: 64 chunks × 2,048 tokens of the calibration text, no output layer | new: all 314 chunks × 2,048 tokens of the same text, with --process-output and --parse-special. The file headers count twice the chunks in this fork (128 for v1, 628 for v3). |
All other explicit type assignments are the same as v2. The full per-tensor arguments are in recipes/8GB-v3.txt.
Where the gain comes from. It comes mostly from the recipe, only a little from the new imatrix. An intermediate recipe (v3 without the blocks 12–19 change) was built with both imatrices. With v1's imatrix it reached 0.0592 / 0.0306 / 0.0368 on sets 1 / 3 / 4, against v2's 0.0703 / 0.0359 / 0.0425. The new imatrix moved that to 0.0586 / 0.0306 / 0.0347 (0 to 6% more). On set 2, which partly overlaps the calibration text, it gave 12% (0.0721 → 0.0634).
MTP at Q4_K. The MTP block has no imatrix data, so it is plain Q4_K. (llama-imatrix --nextn stopped with an assertion in this fork; evals/v3-imx-nextn-crash.log.) Q4_K freed 67.6 MiB for the main model (evals/v3-deney1.log). On the code prompt below, acceptance went from 71.9% (v2) to 69.0% (v3). This was measured on one prompt only.
Each candidate change was spliced into a base file, one at a time, and scored on the sets above. The spliced tensors came from a single-type GGUF quantized from the same BF16. The logs are in evals/ (v3-deney1.csv, v3-deney2.csv). The main findings:
attn_qkv/attn_gate/ssm_out at Q5_K gave about −5% on average over sets 1–3. It was the best of eleven candidates tried.ffn_down at Q5_K, another 1–3% mean KLD (but a higher 99th percentile on set 2). With it, shared memory reached 244 MiB at 104,448 (text) and 222 MiB at 57,344 (images). The largest contexts that fit dropped to 100,352 and 53,248, so it was left out.RTX 4060 8GB, Windows, no display, single runs. The same prompt as in v2 (58 tokens) asks for a Python function plus three pytest tests. Output was 400 tokens, temperature 0, thinking off, --spec-draft-n-max 2.
Configuration (-c) |
MTP off | MTP on | Drafted tokens accepted |
|---|---|---|---|
| v3 (104,448) | 25.4 tok/s | 47.0 tok/s | 231 / 335 (69.0%) |
| v2 8GB-v2 (104,448), from the v2 card | 25.9 tok/s | 47.8 tok/s | 235 / 327 (71.9%) |
v3 measured 1.7% (MTP on) and 2.1% (MTP off) below the v2 card. Over several v3 test builds the same test gave 46–49 tok/s, so treat the speed as about the same.
Long context, MTP on. Three facts were hidden in long text near the limit. The model was then asked to list them:
Configuration (-c) |
Prompt length | Found | Prefill | Generation at that length |
|---|---|---|---|---|
| v3 (104,448) | 102,903 tokens | 3/3 | 736 tok/s | 21.6 tok/s |
| v3 + mmproj (57,344) | 55,728 tokens | 3/3 | 886 tok/s | 25.4 tok/s |
v3 + the F16 mmproj, -c 57344, MTP on, two synthetic test images (the same as v2):
No vision benchmark was run.
On Windows, the NVIDIA driver can move allocations that do not fit into shared system memory instead of failing. The model then still runs, only slower. Each configuration was started with llama-server, and the per-process GPU Process Memory counters were read after a long-context generation:
| Configuration | Dedicated | Shared |
|---|---|---|
v2 8GB-v2, MTP on, -c 104,448 (re-measured after the long test, evals/v3-adim2.log; the v2 card lists 7,833 MiB after a short generation) |
7,851 MiB | 180 MiB |
| v3, MTP on, -c 104,448 | 7,903 MiB | 180 MiB |
| v3 + mmproj, MTP on, -c 57,344 | 7,949 MiB | 174 MiB |
A context counted only if shared memory stayed at or below 200 MiB. One test build with 48 MiB more weights reached 244 MiB at 104,448, and generation at 102K tokens dropped from about 22 to 6.9 tok/s. So v3 sits close to the edge of this card. With a display attached, the desktop takes VRAM, so use a smaller -c and check "Shared GPU memory" in Task Manager. This was not measured for v3. In v1 a simulated 700 MiB desktop cost about 36K tokens of context. If the same holds for v3 (not tested), that would mean roughly 68K for text and 21K with the mmproj.
The same build as v2:
git clone -c core.longpaths=true -b sensenova-mtp-mhc-cuda \
https://github.com/mission-deny-the-mission/llama.cpp-sensenova-loop
cd llama.cpp-sensenova-loop
git checkout cd78d55
cmake -B build -DGGML_CUDA=ON -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --target llama-server llama-cli
On Windows, the build used for this card ran in "x64 Native Tools Command Prompt for VS 2022" (with the CUDA Toolkit, CMake and Ninja installed):
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --target llama-server llama-cli
-DLLAMA_USE_PREBUILT_UI=OFF matters: without it the build downloads prebuilt web-UI assets, and on 2026-11-19 that download did not match this commit, so the build failed. With the default Visual Studio generator, add --config Release to the build command; the binaries are then in build\bin\Release\. Clone into a short path (for example C:\sl). The repository has very long file names, and a clone under a deep folder can silently skip files.
# text only
hf download drofn/SenseNova-Loop-9B-8GB-v3-GGUF SenseNova-Loop-9B-8GB-v3.gguf --local-dir ./models
# text + images
hf download drofn/SenseNova-Loop-9B-8GB-v3-GGUF SenseNova-Loop-9B-8GB-v3.gguf mmproj-SenseNova-Loop-9B-F16.gguf --local-dir ./models
Text, MTP on:
./build/bin/llama-server -m ./models/SenseNova-Loop-9B-8GB-v3.gguf -ngl 99 -fa on \
-ctk q4_0 -ctv q4_0 -c 104448 -ub 128 -np 1 --jinja --reasoning-format deepseek --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 2 -ctkd q4_0 -ctvd q4_0
Text + images, MTP on:
./build/bin/llama-server -m ./models/SenseNova-Loop-9B-8GB-v3.gguf \
--mmproj ./models/mmproj-SenseNova-Loop-9B-F16.gguf -ngl 99 -fa on \
-ctk q4_0 -ctv q4_0 -c 57344 -ub 128 -np 1 --jinja --reasoning-format deepseek --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 2 -ctkd q4_0 -ctvd q4_0
--spec-type draft-mtp --spec-draft-n-max 2 -ctkd q4_0 -ctvd q4_0.image_url content part of /v1/chat/completions (base64 data URIs work).Base weights and BF16. The same BF16 GGUF as v2, with the MTP head (sensenova/SenseNova-Loop-9B as of 2026-11-18; converter patch in patches/, index helper in tools/).
To re-create the files, also build the llama-imatrix and llama-quantize targets.
imatrix. llama-imatrix -m <Q8_0 GGUF> -f calib2.txt -c 2048 -fa on -fit on --process-output --parse-special -o SenseNova-Loop-9B-v3.imatrix.gguf over all 314 chunks of the same calibration text as v1 (v1 used the first 64) (the run named its output imx3.gguf; evals/v3-imx.ps1). Adding --nextn (MTP data) stopped with an assertion in this fork, so the MTP block has no imatrix data.
Quantization. The recipe file holds quoted arguments, so let the shell parse it, for example in bash:
eval "./build/bin/llama-quantize --imatrix SenseNova-Loop-9B-v3.imatrix.gguf $(cat recipes/8GB-v3.txt) BF16.gguf SenseNova-Loop-9B-8GB-v3.gguf Q4_K_M"
(the published file was made with the same arguments from PowerShell; evals/v3-final2.ps1).
Context limits. Each limit was found with llama-server at the listed -c, with MTP (and the mmproj for images). A value counted only if a long generation succeeded and shared GPU memory stayed at or below 200 MiB.
cd78d55 above). No other runtime was tried with v3; v1 and v2 could not be loaded by stock llama.cpp at the time.SenseNova-Loop-9B-8GB-v3.gguf: the quantized model with the MTP head. Its internal test name was v3g, and the logs use that name.mmproj-SenseNova-Loop-9B-F16.gguf: SenseNova's official vision projector, unchanged.SenseNova-Loop-9B-8GB-v3-HQ.gguf (added 2026-11-20): the HQ variant, also with the MTP head. Its internal test name was v31hq3. The published file was re-quantized so its header stores the imatrix as imx3.gguf instead of a local path; its tensor data is byte-identical to the measured file (evals/v3-HQ-repack.log).recipes/8GB-v3.txt: the full llama-quantize tensor-type arguments. recipes/8GB-v3-HQ.txt: the same for HQ.imatrix/SenseNova-Loop-9B-v3.imatrix.gguf: the new importance matrix. Inside the file it is called imx3.gguf.patches/v2-converter.patch, tools/idx_mtp.py: needed to re-create the BF16 GGUF with MTP (unchanged from v2). idx_mtp.py rewrites the index file in place, so run it on a copy of the base model folder.evals/: the measurement scripts and their raw logs. Local paths are shortened to C:\ws. needle_client.py is the needle-test client (called final_test.py inside the scripts). ekle.py is the tensor-splicing tool used for the one-change-at-a-time tests. The scripts use fixed paths and delete their own intermediate GGUFs, so read them before running them anywhere. sizes.txt lists the file sizes of the test builds. v3-HQ-* are the HQ measurements; their scripts also hold steps for other candidates that were not published.GGUF quants sized for one GPU, each with KLD, VRAM and speed measured on the same rig.
| GPU | Model | Repo |
|---|---|---|
| 16 GB | Inkling-2-Mini, 64K / 96K / 128K context | drofn/Inkling-2-Mini-16GB-GGUF |
| 8 GB | SenseNova-Loop-9B, recommended build | drofn/SenseNova-Loop-9B-8GB-v3-GGUF |
| 6 GB | SenseNova-Loop-9B, 32K context | drofn/SenseNova-Loop-9B-6GB-GGUF |
| any | SenseNova-Loop-9B, full ladder in the upstream llama.cpp layout | drofn/SenseNova-Loop-9B-GGUF |
Earlier 8 GB builds, kept for reference: v2 and v1.
Apache-2.0, the same as the base model.