Newer version: SenseNova-Loop-9B-8GB-v3-GGUF is the recommended 8 GB build. Its mean KLD vs Q8_0 is about 20% lower than this repo's 8GB-100K on all four test sets in the v3 card (8GB-v2 uses the same weights), at 104,448 tokens of context, with the MTP head (about 1.85x faster generation) and optional image input. This repo stays up for the 128K and 32K-HQ variants. All sizes and versions: collection.
Text-only GGUF quantizations of SenseNova-Loop-9B, with the bit budget allocated per tensor group from measured sensitivity, aimed at long-context inference on 8 GB NVIDIA cards.
In short: the 8GB-100K file is smaller than a standard Q4_K_M made with the same imatrix, fits ~10K more context on an 8 GB card by llama-fit-params (104,704 vs 94,464 tokens), and has 34–38% lower KL divergence from Q8_0 (0.0703 / 0.0762 vs 0.1064 / 0.1235). Everything runs on the GPU; only the token embedding table stays in system RAM.
| File | Size | Fits on 8 GB (no display, llama-fit-params) |
Quality (mean KLD vs Q8_0) | Pick it if |
|---|---|---|---|---|
| SenseNova-Loop-9B-8GB-128K.gguf | 4.97 GB | 132,864 tokens | 0.1375 / 0.1213 | you need the longest context |
| SenseNova-Loop-9B-8GB-100K.gguf | 5.54 GB | 104,704 tokens | 0.0703 / 0.0762 | best balance in this repo, lower KLD than Q4_K_M (for a new setup, see v3 below) |
| SenseNova-Loop-9B-8GB-32K-HQ.gguf | 7.02 GB | 38,400 tokens | 0.0182 / 0.0245 | you want the closest match to Q8_0 and 32K is enough (GPU without a display) |
KLD values are set 1 / set 2 (see "Quality comparison"). Lower is better. The context values are llama-fit-params estimates; the long runs under "Speed and VRAM" used 131,072 / 104,448 / 38,400.
Patched llama.cpp required. Upstream llama.cpp, LM Studio and Ollama did not support the
sensenova_looparchitecture as of 2026-11-19 (ggml-org/llama.cpp#31571, open then). Build the fork listed under "How to run".
Newer: v3 (about 20% lower KLD than v2's 8GB-v2 at the same 104,448 context, MTP head, optional image input) and v2 (this repo's mixes plus SenseNova's MTP head, about 1.85x faster generation on an RTX 4060 with --spec-type draft-mtp, and an 8GB-v2-Vision file). On the same test stand, AtomicChat's AD-Q4_K_M is as close to Q8_0 as v3 or closer; see the v3 card.
Have a 6 GB card? See drofn/SenseNova-Loop-9B-6GB-GGUF: one 32K-context file sized for 6 GB GPUs, built with the same method.
KV cache follow-up (measured on the 6 GB file, same architecture): at ~31K tokens, the retrieval tasks (6 haystacks each) found every code with f16, q8_0 and q4_0 KV; K q8_0 / V q4_0 was run on one of them and also found every code. Following six formatting rules given far back in the system prompt passed 13/18 haystacks with f16, 12/18 with q8_0 and 9/18 with q4_0, a hint (not a proven difference) that q4_0 KV costs some instruction following at long context. Details and raw results are on the 6 GB card.
SenseNova-Loop-9B is a ~9B decoder with a looped layer block. It stores 32 decoder layers and runs layers 4–27 twice per token, giving 56 layer executions per forward pass. Its residual stream uses two-stream manifold-constrained hyper-connections (mHC), with routing normalized by 20 Sinkhorn iterations. Attention is hybrid: each group of four layers contains three linear-attention (Gated DeltaNet) layers and one full-attention layer. Full attention uses 16 query heads, 4 KV heads and head_dim 256. The vocabulary has 248,320 tokens, and the context length is 512K (YaRN x2).
The original model also accepts images. These GGUFs are text-only and contain no vision projector (mmproj).
Context: all layers on the GPU, q4_0 KV cache, RTX 4060 8GB under Windows with no display attached. With a display attached, see "Which -c to use".
| Variant | File | Size (GB) | Avg bpw | Context: llama-fit-params estimate / longest run |
|---|---|---|---|---|
| 8GB-128K | SenseNova-Loop-9B-8GB-128K.gguf |
4.97 | 4.44 | 132,864 / 131,072 |
| 8GB-100K | SenseNova-Loop-9B-8GB-100K.gguf |
5.54 | 4.95 | 104,704 / 104,448 |
| 8GB-32K-HQ | SenseNova-Loop-9B-8GB-32K-HQ.gguf |
7.02 | 6.27 | 38,400 / 38,400 |
Choose a variant by how much context you need, then use the "Which -c to use" table to pick a starting -c value.
All rows use the same imatrix. KLD, same-top-p and PPL are measured against the Q8_0 reference, which is not published. Lower KLD is closer to Q8_0. These metrics do not measure task accuracy.
| Quant | Size (GB) | Avg bpw | Mean KLD vs Q8_0 (set 1 / set 2) | Same top token % (set 1 / set 2) | PPL (set 1 / set 2) |
|---|---|---|---|---|---|
| Q8_0 (reference, not published) | 9.53 | 8.51 | 0 | 100 | 11.555 / 2.633 |
| 8GB-128K | 4.97 | 4.44 | 0.1375 / 0.1213 | 82.6 / 92.0 | 12.266 / 2.682 |
| 8GB-100K | 5.54 | 4.95 | 0.0703 / 0.0762 | 88.5 / 94.5 | 11.584 / 2.640 |
| 8GB-32K-HQ | 7.02 | 6.27 | 0.0182 / 0.0245 | 93.7 / 97.2 | 11.694 / 2.650 |
| Standard Q4_K_M (same imatrix) | 5.62 | 5.02 | 0.1064 / 0.1235 | 86.4 / 93.5 | 11.795 / 2.736 |
Metric definitions and evaluation text: two held-out sets, each scored with llama-perplexity --kl-divergence over 40 chunks of 512 tokens against Q8_0 logits: set 1 = Turkish and English Wikipedia prose; set 2 = English Wikipedia and tool-calling dialogues (glaive-function-calling-v2). The 40 scored chunks cover the first 39% of the set-2 file; the Python source at its end was not reached, so code is not part of these numbers. Values are shown as set 1 / set 2.
KLD is the primary metric here: it compares the full next-token distribution with Q8_0. PPL on these short sets carries a statistical uncertainty of roughly ±3% (as reported by llama-perplexity: 11.55 ± 0.36 for Q8_0 on set 1), so the PPL differences in the table above are not a reliable ranking.
Measured on an RTX 4060 8GB (Windows, no display attached), each variant started at its longest tested context (131,072 / 104,448 / 38,400) with the command under "How to run". All speed figures are single runs (not medians of repeated runs), with thinking on; "short prompt" is a ~150-word writing request, so expect a few percent of run-to-run variation. The needle test hides three code words at 10%, 50% and 90% depth of a long Turkish document and asks for all three.
Variant (-c) |
Generation, short prompt | Needle test near the limit | Generation at that length | Prefill at that length |
|---|---|---|---|---|
| 8GB-128K (131,072) | 28.3 tok/s | 129,575 tokens: 3/3 found | 16.5 tok/s | 702 tok/s |
| 8GB-100K (104,448) | 25.9 tok/s | 102,924 tokens: 3/3 found | 17.0 tok/s | 772 tok/s |
| 8GB-32K-HQ (38,400) | 21.0 tok/s | 36,903 tokens: 3/3 found | 18.2 tok/s | 964 tok/s |
VRAM in use at the longest tested context (nvidia-smi): 6,879–6,901 MiB of 8,188 MiB for 8GB-100K and 6,895 MiB for 8GB-32K-HQ; the rest is held by the Windows driver plus the 256 MiB safety margin. 8GB-128K was sized with the same margin but its VRAM was not read separately. System RAM (working set) of the server process was 1.2–1.5 GB in the 100K runs: the token embedding table plus the CUDA runtime. Multi-turn: with --ctx-checkpoints 4 the server keeps the conversation state on the GPU, so follow-up turns do not re-read the whole context. Asking again over the same 50K-token context took 54 s the first time and 2.1 s the second time (measured during development with the same server settings; this hybrid model cannot reuse its cache without these checkpoints).
A four-question smoke test (geography, arithmetic, a VAT calculation, a Python one-liner, with thinking on) was answered correctly by all three variants.
These are single-machine figures from one GPU. Other 8 GB cards, drivers and operating systems may give different numbers.
The usable context depends on how much VRAM is left after the desktop and other applications. Start with these values, and reduce -c if you run out of memory.
| Setup | 8GB-128K | 8GB-100K | 8GB-32K-HQ |
|---|---|---|---|
| Windows, no display attached | 132,864 | 104,704 | 38,400 |
| Windows, display attached (~700 MiB used by the desktop, simulated) | 96,768 | 68,608 | does not fit fully; use 8GB-100K |
Both rows come from llama-fit-params on the RTX 4060 (q4_0 KV, -ub 128, 256 MiB safety margin; the display row adds 700 MiB to the margin). Linux was not measured; it usually leaves a few hundred MiB more VRAM free than Windows, so the Windows numbers are a safe starting point.
If you would rather not set -c by hand, omit it and add -fit on, which lets llama.cpp adjust unset arguments to fit device memory. Check the startup log for the context size it selected.
git clone -c core.longpaths=true -b sensenova-mtp-mhc-cuda \
https://github.com/mission-deny-the-mission/llama.cpp-sensenova-loop
cd llama.cpp-sensenova-loop
git checkout cd78d55
cmake -B build -DGGML_CUDA=ON -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --target llama-server llama-cli
This commit is the head of GuideboardLabs/llama.cpp-sensenova-loop PR #1. It adds CUDA kernels for the mHC routing ops. Without them, those ops run on the CPU in CUDA builds.
hf download drofn/SenseNova-Loop-9B-8GB-GGUF SenseNova-Loop-9B-8GB-128K.gguf --local-dir ./models
./build/bin/llama-server -m ./models/SenseNova-Loop-9B-8GB-128K.gguf -ngl 99 -fa on \
-ctk q4_0 -ctv q4_0 -c 131072 -ub 128 -np 1 --jinja \
--reasoning-format deepseek --cache-ram 0 --ctx-checkpoints 4
On Windows, the build used for these files ran in "x64 Native Tools Command Prompt for VS 2022" (with the CUDA Toolkit, CMake and Ninja installed):
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --target llama-server llama-cli
-DLLAMA_USE_PREBUILT_UI=OFF matters: without it the build downloads prebuilt web-UI assets, and on 2026-11-19 that download did not match what this commit expects, so the build failed. With it, llama-server has no built-in web page but serves the OpenAI-compatible API as usual. The binaries are then in build\bin\. With the default Visual Studio generator instead, add --config Release to the build command (otherwise you get a slow Debug build); the binaries are then in build\bin\Release\. Clone into a short path (for example C:\sl): the repository contains very long file names, and a clone under a deep folder can silently skip files. Building is only needed for inference; the converter patches in patches/ matter only if you want to re-create the GGUFs from the safetensors.
-ngl 99: offload all layers to the GPU.-fa on: flash attention, which the quantized V cache requires.-ctk q4_0 -ctv q4_0: q4_0 KV cache for K and V.-c 131072: context size; for the other files and setups, take the value from the "Which -c to use" table.-ub 128: micro-batch size.-np 1: one parallel slot.--jinja: use the chat template embedded in the GGUF.--reasoning-format deepseek: return reasoning text separately from the answer.--cache-ram 0, --ctx-checkpoints 4: prompt-cache settings used in the measured runs.Test request:
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "What is 13 times 7?"}]}'
Calibration. The importance matrix (imatrix) was computed on Turkish, English, code and tool-calling text, rendered with the model's chat template where it applies. The calibration file is about 2.5 MB (314 chunks of 2,048 tokens), shuffled so every chunk type is spread through it; the imatrix run used its first 64 chunks (131,072 tokens) and was made without --process-output, so it has no statistics for the output head. The file header says 128 chunks because this fork counts each chunk twice. v3 later used all 314 chunks. The same imatrix was used for every published quant and for the Q4_K_M comparison row. It is included in imatrix/.
Sensitivity sweep. Pass 1 split the tensors into 14 groups: FFN gate/up, FFN down, linear attention and full attention, each divided into input layers (0–3), loop layers (4–27) and output layers (28–31), plus the output head and the token embedding. Pass 2 split the loop block into three bands (4–11, 12–19, 20–27) for FFN and linear attention. Each group was quantized on its own to Q2_K and to Q4_K, with everything else at Q8_0, and the KL divergence against Q8_0 was measured (28 runs in pass 1, 12 in pass 2). Pass 1 used an earlier imatrix (512-token chunks); pass 2 used the final one.
Allocation. For each group, KLD was modelled as an exponential in bits per weight through the two measured points, with Q8_0 as zero. A greedy search then upgraded the group with the best KLD reduction per extra byte until the VRAM budget for the variant's context was used. The budget was found with llama-fit-params on the target card. The exact per-tensor types for each file are in recipes/.
Sweep results:
Each row: only that group quantized (everything else Q8_0), mean KLD vs Q8_0 on set 1. Loop-band rows come from a second, finer sweep with the final imatrix.
| Tensor group | KLD at Q2_K | KLD at Q4_K | KLD per GB saved (Q2_K) |
|---|---|---|---|
| FFN gate+up, layers 0-3 | 0.1056 | 0.0154 | 0.36 |
| FFN down, layers 0-3 | 0.0871 | 0.0136 | 0.59 |
| Linear attention, layers 0-3 | 0.1657 | 0.0546 | 1.12 |
| Full attention, layer 3 | 0.0335 | 0.0058 | 0.78 |
| FFN (all), loop layers 4-11 | 0.1808 | 0.0270 | 0.20 |
| FFN (all), loop layers 12-19 | 0.0944 | 0.0108 | 0.11 |
| FFN (all), loop layers 20-27 | 0.1282 | 0.0114 | 0.14 |
| Linear attention, loop 4-11 | 0.0869 | 0.0143 | 0.29 |
| Linear attention, loop 12-19 | 0.0504 | 0.0066 | 0.17 |
| Linear attention, loop 20-27 | 0.0274 | 0.0032 | 0.09 |
| Full attention, loop layers | 0.0731 | 0.0093 | 0.28 |
| FFN gate+up, layers 28-31 | 0.0599 | 0.0046 | 0.20 |
| FFN down, layers 28-31 | 0.0227 | 0.0022 | 0.15 |
| Linear attention, layers 28-31 | 0.0123 | 0.0013 | 0.08 |
| Full attention, layer 31 | 0.0082 | 0.0009 | 0.19 |
| Output head | 0.0956 | 0.0057 | 0.13 |
| Token embedding (CPU) | 0.0835 | 0.0136 | 0.11 |
Key findings:
loop_gate.logits tensor is skipped (the reference modeling code does not use it), MTP is disabled (no_mtp=True; the converter used then did not convert the release's MTP head, which v2 later added), and the mHC router weights stay in F32. The patches are in patches/.SenseNova-Loop-9B-8GB-128K.gguf, SenseNova-Loop-9B-8GB-100K.gguf, SenseNova-Loop-9B-8GB-32K-HQ.gguf: quantized models (see Variants).imatrix/SenseNova-Loop-9B.imatrix.gguf: importance matrix used for all quants.recipes/*.txt: per-group quantization recipe for each variant.patches/: one patch file with the three converter changes described under Limitations (apply with git apply on top of cd78d55).LICENSE: Apache License 2.0 text (the base model is released under Apache-2.0).Apache-2.0, the same as the base model. The quantized files are derivatives of SenseNova-Loop-9B.