Text-only GGUF quantizations of SenseNova-Loop-9B, with the bit budget allocated per tensor group from measured sensitivity, aimed at 32K-context inference on 6 GB NVIDIA cards (RTX 2060 6GB, RTX 3050 6GB and similar). This is the 6 GB sibling of drofn/SenseNova-Loop-9B-8GB-GGUF.
In short: the 6GB-32K file (4.72 GB; the standard quants below are 4.24–4.63 GB) has 41–67% lower KL divergence from Q8_0 than the standard IQ3_M, Q3_K_M, IQ3_XS, Q3_K_S quants made with the same imatrix on set 1 (0.168 vs 0.288 / 0.285 / 0.288 / 0.504), 24–39% lower on set 2 and 23–53% lower on set 3, and fits 39,680 tokens of context with q4_0 KV on a 6 GB card with no display, against 40,192, 30,464, 49,920, 48,896 for IQ3_M, Q3_K_M, IQ3_XS, Q3_K_S. Everything runs on the GPU; only the token embedding table stays in system RAM. The file reaches 81.7% same-top-token rate on set 1 (our quality bar: at least 80%).
| File | Size | Context, no display (q4_0 / q8_0 KV) | KLD (set 1 / 2 / 3) | Pick it if |
|---|---|---|---|---|
| SenseNova-Loop-9B-6GB-32K.gguf | 4.72 GB | 39,680 / 23,296 | 0.168 / 0.146 / 0.090 | recommended: main file, 32K context with q4_0 KV |
KLD values are set 1 / set 2 / set 3 (see "Quality comparison"). Lower is better. "Context" is the maximum that fits on a 6 GB card with no display attached (simulated, see "Which -c to use").
Quality bar. We require a same-top-token rate of at least 80% on set 1 (how often the quant picks the same top token as Q8_0). Measured values: 6GB-32K 81.7%, Standard IQ3_M (same imatrix) 77.7%, Standard Q3_K_M (same imatrix) 78.3%, Standard IQ3_XS (same imatrix) 78.0%, Standard Q3_K_S (same imatrix) 72.1%. Meets the bar: 6GB-32K. Below the bar: Standard IQ3_M (same imatrix), Standard Q3_K_M (same imatrix), Standard IQ3_XS (same imatrix), Standard Q3_K_S (same imatrix). A 3.6 bpw 64K variant was also built (set 1: 0.333 mean KLD, 74.8% same top) but it is below the bar and did not beat the standard IQ3_XS-imatrix quant of similar size (0.288), so it is not published.
Patched llama.cpp required. Upstream llama.cpp, LM Studio and Ollama do not support the
sensenova_looparchitecture yet (ggml-org/llama.cpp#31571, open). Build the fork listed under "How to run".
Have an 8 GB card? See drofn/SenseNova-Loop-9B-8GB-GGUF: it has variants that trade quality against context for 8 GB cards, including longer-context ones.
SenseNova-Loop-9B is a ~9B decoder with a looped layer block. It stores 32 decoder layers and runs layers 4–27 twice per token, giving 56 layer executions per forward pass. Its residual stream uses two-stream manifold-constrained hyper-connections (mHC), with routing normalized by 20 Sinkhorn iterations. Attention is hybrid: each group of four layers contains three linear-attention (Gated DeltaNet) layers and one full-attention layer. Full attention uses 16 query heads, 4 KV heads and head_dim 256. The vocabulary has 248,320 tokens, and the context length is 512K (YaRN x2).
The original model also accepts images. These GGUFs are text-only and contain no vision projector (mmproj).
iq2_s, iq3_xxs, iq3_s and iq4_xs were added to the allocator's menu. At the 32K target, the IQ menu reached 0.187 mean KLD on set 1 (40-chunk run, before the final budget resize), against 0.223 for a K-quant-only menu and 0.197 for a mixed K-quant and IQ menu.| Variant | File | Size (GB) | Avg bpw | Max context, no display (q4_0 / q8_0 KV) |
|---|---|---|---|---|
| 6GB-32K | SenseNova-Loop-9B-6GB-32K.gguf |
4.72 | 4.20 | 39,680 / 23,296 |
Max context values are llama-fit-params estimates on a simulated 6 GB card (see "Which -c to use"), not runs on a real 6 GB GPU.
All rows use the same imatrix. KLD and same-top-token rate are measured against the Q8_0 reference, which is not published. Lower KLD is closer to Q8_0; higher same-top is better. These metrics do not measure task accuracy.
| Quant | Size (GB) | Avg bpw | Mean KLD vs Q8_0 (set 1 / 2 / 3) | Same top token % (set 1 / 2 / 3) | 99th-percentile KLD (set 1) |
|---|---|---|---|---|---|
| Q8_0 (reference, not published) | 9.53 | 8.51 | 0 | 100 | 0 |
| 6GB-32K | 4.72 | 4.20 | 0.168 / 0.146 / 0.090 | 81.7 / 90.9 / 89.6 | 1.369 |
| Standard IQ3_M (same imatrix) | 4.44 | 3.96 | 0.288 / 0.197 / 0.117 | 77.7 / 89.8 / 88.7 | 2.871 |
| Standard Q3_K_M (same imatrix) | 4.63 | 4.14 | 0.285 / 0.191 / 0.124 | 78.3 / 90.4 / 89.0 | 2.953 |
| Standard IQ3_XS (same imatrix) | 4.24 | 3.79 | 0.288 / 0.197 / 0.129 | 78.0 / 89.9 / 88.3 | 2.749 |
| Standard Q3_K_S (same imatrix) | 4.26 | 3.81 | 0.504 / 0.240 / 0.194 | 72.1 / 88.7 / 86.3 | 4.943 |
The 99th-percentile KLD column shows the tail of the per-token KLD distribution on set 1 (same run as the set-1 columns). File sizes include the token embedding table (1.02B parameters), which stays in system RAM and is not loaded onto the GPU.
Metric definitions and evaluation text: three held-out sets, each scored with llama-perplexity --kl-divergence over 40 chunks of 512 tokens against Q8_0 logits (set definitions under "Method"). Values are shown as set 1 / set 2 / set 3.
KLD is the primary metric here: it compares the full next-token distribution with Q8_0.
The recommended setup is a quantized KV cache, so we measured what it costs. The test uses one 32K-token document (the llama.cpp C++ source) and compares each KV type against a bf16 KV cache.
| KV type | KLD vs bf16 KV | Over f16 floor | Same top token % | PPL |
|---|---|---|---|---|
| f16 | 0.282 | — | 94.24 | 5.53 |
| q8_0 | 0.289 | +0.007 | 93.96 | 5.47 |
| K q8_0 / V q4_0 | 0.403 | +0.121 | 92.69 | 5.57 |
| K q4_0 / V q8_0 | 0.474 | +0.192 | 92.07 | 5.36 |
| q4_0 | 0.505 | +0.223 | 91.42 | 5.44 |
Even f16 differs from bf16 by 0.282 KLD, so about 0.28 is the noise floor of this test. q8_0 is 0.007 above that floor, effectively equal to it; q4_0 adds about 0.22 KLD. With one document the numbers can be noisy, so read them as a rough guide. On a 6 GB card with no display, 6GB-32K fits 23,296 tokens with q8_0 KV and 39,680 with q4_0 KV. If you need 32K or more, use q4_0; if about 23K is enough, q8_0 costs less quality. Both are valid, and you choose.
Speed on an RTX 4060 (6 GB cards may be slower; measured with the 8GB-100K file from the 8 GB repo, llama-bench, 3 repetitions) does not depend much on the KV type:
| KV type | pp512 (tok/s) | tg128 (tok/s) | pp512 at 16K depth | tg128 at 16K depth |
|---|---|---|---|---|
| f16 | 1088.7 | 24.4 | 865.8 | 22.7 |
| q8_0 | 1072.7 | 24.6 | 901.3 | 22.9 |
| q4_0 | 1054.3 | 24.3 | 913.4 | 22.8 |
pp512 is prefill of 512 tokens and tg128 is generation of 128 tokens, with an empty context and at 16K depth. Between KV types the spread is at most 1.3% in generation (empty context), 0.7% at 16K depth, and 3.2% / 5.2% in prefill, so on this GPU the KV type is a memory and quality choice, not a speed choice.
To see where the KV-cache loss actually shows up, we filled ~31K-token prompts with synthetic warehouse records and ran the 6GB-32K file with four KV cache types. Temperature 0, thinking off, RTX 4060; n is the number of different haystacks.
| Task (~31K-token prompt) | f16 | q8_0 | K q8_0 / V q4_0 | q4_0 |
|---|---|---|---|---|
| Recall 5 codes, free format (n=6) | 100% | 100% | not run | 100% |
| Recall 5 codes as strict JSON (n=6) | 6/6 | 6/6 | not run | 6/6 |
| Pick 3 codes out of 20 similar ones (n=6) | 100% | 100% | 100% | 100% |
| 3 format rules just before the answer (n=6) | 6/6 | 6/6 | not run | 6/6 |
| 3 format rules in the system prompt (n=6) | 5/6 | 5/6 | not run | 6/6 |
| 6 format rules in the system prompt (n=18) | 13/18 | 12/18 | 9/18 | 9/18 |
-ctk q8_0 -ctv q4_0), but on the 6-rule test it scored the same as q4_0.Scripts and raw per-haystack results are in evals/.
Speed figures are measured on an RTX 4060 (8 GB). 6 GB cards such as the RTX 2060 or RTX 3050 6GB are often slower, so treat these as an upper reference, not a promise.
-c 32768): 3/3 codes returned in full when the prompt asks for the full codes. With the shorter prompt used on the 8 GB card ("give only the words"), the model answered with the first word of each code (correct words, correct order), which our strict full-code check scores as 0/3. With q8_0 KV (-c 23040, 21,993-token prompt): 3/3.VRAM, measured on the RTX 4060 with nvidia-smi (llama-server, -ngl 99 -fa on -ub 128 -np 1): 4,689 MiB after loading with -c 32768 and q4_0 KV, 4,713 MiB after the 31K-token prompt; 4,813 MiB with -c 23040 and q8_0 KV. A headless 6 GB card has roughly 5,060 MiB usable after the driver reserve (the 4060 reserves about 1,080 MiB with no display attached), so both settings fit, the q8_0 one with little margin.
4 GB cards: with partial CPU offload (-fit on, 4 GB card simulated on the 4060, 32K q4_0) generation was 2.5 tok/s, so we do not recommend this file for 4 GB cards.
These are single-machine figures from one GPU. Other cards, drivers and operating systems may give different numbers.
The usable context depends on how much VRAM is left after the desktop and other applications. Start with these values, and reduce -c if you run out of memory.
| Setup | 6GB-32K |
|---|---|
| No display, q4_0 KV | 39,680 |
| No display, q8_0 KV | 23,296 |
| Display attached (~700 MiB used by the desktop, assumed), q4_0 KV | 3,584 |
| Display attached (~700 MiB used by the desktop, assumed), q8_0 KV | 2,048 |
With a display attached, 6GB-32K reaches only 3,584 tokens with q4_0 KV in this simulation, which is below 32K.
All rows come from llama-fit-params run on an RTX 4060 (8 GB). With no display, the 4060 has about 7,106 MiB free after the driver reserve (about 1,080 MiB); a 6 GB card with the same reserve has about 5,060 MiB. The extra 2,044 MiB plus a 256 MiB safety margin were held back with -fitt 2300 (-ngl 99 -fa on -np 1 -ub 128). The display row is an assumption, not a measurement: it adds 700 MiB for a desktop session. Real 6 GB cards were not tested; actual free VRAM depends on the driver, operating system and what else is running. "—" means the file does not fit even 2,048 tokens of context in that setup.
If you would rather not set -c by hand, omit it and add -fit on, which lets llama.cpp adjust unset arguments to fit device memory. Check the startup log for the context size it selected.
git clone -c core.longpaths=true -b sensenova-mtp-mhc-cuda \
https://github.com/mission-deny-the-mission/llama.cpp-sensenova-loop
cd llama.cpp-sensenova-loop
git checkout cd78d55
cmake -B build -DGGML_CUDA=ON -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --target llama-server llama-cli
This commit is the head of GuideboardLabs/llama.cpp-sensenova-loop PR #1. It adds CUDA kernels for the mHC routing ops. Without them, those ops run on the CPU in CUDA builds.
hf download drofn/SenseNova-Loop-9B-6GB-GGUF SenseNova-Loop-9B-6GB-32K.gguf --local-dir ./models
./build/bin/llama-server -m ./models/SenseNova-Loop-9B-6GB-32K.gguf -ngl 99 -fa on \
-ctk q4_0 -ctv q4_0 -c 32768 -ub 128 -np 1 --jinja \
--reasoning-format deepseek --cache-ram 0 --ctx-checkpoints 4
On Windows, the build used for these files ran in "x64 Native Tools Command Prompt for VS 2022" (with the CUDA Toolkit, CMake and Ninja installed):
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=OFF
cmake --build build --target llama-server llama-cli
-DLLAMA_USE_PREBUILT_UI=OFF matters: without it the build downloads prebuilt web-UI assets, and on 2026-11-19 that download did not match what this commit expects, so the build failed. With it, llama-server has no built-in web page but serves the OpenAI-compatible API as usual. The binaries are then in build\bin\. With the default Visual Studio generator instead, add --config Release to the build command (otherwise you get a slow Debug build); the binaries are then in build\bin\Release\. Clone into a short path (for example C:\sl): the repository contains very long file names, and a clone under a deep folder can silently skip files. Building is only needed for inference; the converter patches in patches/ matter only if you want to re-create the GGUFs from the safetensors.
-ngl 99: offload all layers to the GPU.-fa on: flash attention, which the quantized V cache requires.-ctk q4_0 -ctv q4_0: q4_0 KV cache for K and V. Use -ctk q8_0 -ctv q8_0 for the higher-quality KV cache, and lower -c to the q8_0 value from "Which -c to use".-c <ctx>: context size, from the "Which -c to use" table.-ub 128: micro-batch size.-np 1: one parallel slot.--jinja: use the chat template embedded in the GGUF.--reasoning-format deepseek: return reasoning text separately from the answer.--cache-ram 0, --ctx-checkpoints 4: prompt-cache settings from the 8 GB repo's runs. This hybrid model cannot reuse its cache across turns without the checkpoints.Test request:
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "What is 13 times 7?"}]}'
Same method as the 8 GB repo, with one change for 6 GB (the i-quant menu).
Calibration. The importance matrix (imatrix) was computed on Turkish, English, code and tool-calling text, rendered with the model's chat template where it applies: 131,072 tokens (64 chunks of 2,048), about 2.5 MB of text, shuffled so every chunk type is covered. The same imatrix was used for every quant in this repo and for the standard-quant comparison rows. It is included in imatrix/.
Sensitivity sweep. The tensors were split into groups (FFN gate/up, FFN down, linear attention and full attention, each divided into input layers 0–3, loop layers 4–27 and output layers 28–31, plus the output head and the token embedding; the loop block was then split into three bands). Each group was quantized on its own to Q2_K and to Q4_K, with everything else at Q8_0, and the KL divergence against Q8_0 was measured. The per-group table and chart are on the 8 GB card.
Allocation. For each group, KLD was modelled as an exponential in bits per weight through the measured points, with Q8_0 as zero. A greedy search then upgraded the group with the best KLD reduction per extra byte until the VRAM budget for the variant's context was used. For 6 GB the allocator's menu of types included iq2_s, iq3_xxs, iq3_s and iq4_xs. The budget was found with llama-fit-params on a simulated 6 GB card. The exact per-tensor types for each file are in recipes/.
Evaluation sets. Hold-out scoring uses three sets, each over 40 chunks of 512 tokens against Q8_0 logits (llama-perplexity --kl-divergence): set 1 = Turkish and English Wikipedia prose (the 40 scored chunks lie in the prose part at the start of the file); set 2 = English Wikipedia and tool-calling dialogues (glaive-function-calling-v2), where the 40 scored chunks cover the first 39% of the set-2 file and do not reach the Python source at its end; set 3 = code, from the llama.cpp source: Python (conversion/base.py, qwen.py, llama.py) and C++ (src/llama-graph.cpp, src/llama-kv-cache.cpp). Set 3 was not used for calibration.
loop_gate.logits tensor is skipped (the reference modeling code does not use it), MTP is disabled (no_mtp=True, since this release has no MTP), and the mHC router weights stay in F32. The patches are in patches/.SenseNova-Loop-9B-6GB-32K.gguf (main file): quantized models (see Variants).imatrix/SenseNova-Loop-9B.imatrix.gguf: importance matrix used for all quants.recipes/*.txt: per-group quantization recipe for each variant.patches/: one patch file with the three converter changes described under Limitations (apply with git apply on top of cd78d55).evals/: long-context retrieval and instruction-following test scripts (ifl.py, ifl2.py) with raw results (*.jsonl, summary.csv).LICENSE: Apache License 2.0 text (the base model is released under Apache-2.0).SHA-256:
SenseNova-Loop-9B-6GB-32K.gguf: 5096720a718e98bda1e261326c3ffde1414202e85470a189075f3f8f070f27cbApache-2.0, the same as the base model. The quantized files are derivatives of SenseNova-Loop-9B.