Abyss (engine) | Base model | Qwen3.5-9B QSA | QSA idea: higa2515/Qwen3.8-27B-QSA-GGUF | Report a result | 日本語は下
Ornith-1.5-9B QSA GGUF — BETA
- Target: one NVIDIA GPU with 12 GB or more. Measured on an RTX 5090: 128K context in 14.9 GB of VRAM, with 7 GB of host RAM. Not yet measured on this model: a VRAM cap. Our Qwen3.5-9B QSA uses the same tensor shapes and hit the same peaks, and it ran 128K in 10.7 GB with VRAM capped at 11 GB. We expect the same here, but that is an estimate.
- Abyss only. llama.cpp cannot load these files (details below).
ornith-ai/Ornith-1.5-9B with a QSA indexer added to its full-attention layers. On long prompts, each QSA layer reads only the parts of the context the indexer picks. That keeps decoding fast as the context grows. The weights are ornith-ai's, unchanged except for quantization. Only the small indexer (21M parameters) is new.
This is the same recipe as our Qwen3.5-9B QSA: the same architecture, tokenizer, training data and settings. The indexer was trained in 3 h 45 min on one rented RTX 5090, using 2.16 kWh of GPU energy.
BETA: measured on one PC. Read the KL note under Quality. This model's 128K predictions move more than Qwen's, with or without QSA. Reports from other GPUs are welcome (see Feedback).
Model files
| file | quant | size | status |
|---|---|---|---|
Ornith-1.5-9B-QSA-Q4_K_M.gguf |
Q4_K_M + FP32 indexer | 5.85 GB | measured (all numbers below) |
Ornith-1.5-9B-QSA-Q8_0.gguf |
Q8_0 + FP32 indexer | 9.86 GB | measured: 128K quality only |
- Both files keep the model's NextN (MTP) block, so Abyss uses it to draft tokens.
- Chat template: Ornith's own
chat_template.jinja, which renders<think>…</think>on past assistant turns.- Ornith-1.5-9B's
tokenizer_config.jsonstill holds the Qwen3.5 template. A plain llama.cpp conversion would therefore embed Qwen's template. - We embedded Ornith's template instead. Every other key and tensor matches a plain conversion byte for byte.
- Ornith-1.5-9B's
- No imatrix. SHA-256 sums are in
SHA256SUMS.
QSA in brief
- Ornith-1.5-9B has 32 layers, laid out as in Qwen3.5-9B: 24 Gated DeltaNet layers and 8 full-attention layers (3, 7, ..., 31).
- QSA runs in 7 of the 8 full-attention layers. Once the cache passes 4,099 tokens, each token there attends only to
the indexer's 1,024 best 4-token blocks plus its own block (
top_k4096). - Layer 3 stays dense. This is the same choice as on Qwen3.5-9B QSA, where keeping it dense halved the 128K KL. We did not re-tune it for Ornith.
- The file's metadata says this:
attention.compress_ratiosis 0 on layer 3, andattention.indexer.top_kis 4096. - All KV stays in memory. QSA cuts attention work and KV reads, not KV storage.
STRATA_Q35_QSA=0runs the same file with dense attention everywhere.
Run it (Abyss)
You need Abyss v1.3.162 or later, which reads per-layer dense QSA from the file. The one-command setup below comes in v1.3.166. Earlier versions run this file all dense. CUDA only; on AMD it runs dense. An 11 GB+ GPU is expected to hold a 128K context, based on Qwen3.5-9B QSA's measurement.
git clone https://github.com/zephel01/Abyss.git ~/llm/apps/Abyss
cd ~/llm/apps/Abyss
./setup-deepseek.sh --model ornith-9b-qsa --download-model # engine, this file (Q4_K_M), SHA-256 check, pack, config
../Abyss-tools/abyss.sh start # → http://127.0.0.1:8082/
--quant Q8_0takes the Q8_0 file.--context 131072asks for a 128K context.- Windows 11: double-click
setup-deepseek.batwith the same options (setup-deepseek.bat --model ornith-9b-qsa --download-model). Then start it withSTART-ABYSS.bat --config abyss-<name>.json, using the name the setup printed.
The start-up line ends with dense attention layers: 3. Then:
curl http://127.0.0.1:8082/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"What is 17*23?"}],"max_tokens":4000}'
To compare with dense attention, add "env": {"STRATA_Q35_QSA": "0"} to the config.
llama.cpp: not tried with these files. Our Qwen3.5-9B QSA files have the same layout (477 tensors), and llama.cpp
refuses them: wrong number of tensors; expected 477, got 442. Its qwen35 loader does not know the indexer tensors.
For llama.cpp, use a normal Ornith-1.5-9B GGUF.
Quality: QSA vs dense at 128K
The setup:
- Q4_K_M in Abyss, teacher-forced.
- Scored on the last 1,024 tokens of 8 held-out samples of about 128K tokens each. These are the same token files we used for Qwen3.5-9B QSA.
- Dense is the same file with
STRATA_Q35_QSA=0. - nll change: lower is better; 0% means the same as dense.
| sample | dense nll | QSA nll | change | KL | max KL | same top token |
|---|---|---|---|---|---|---|
| code 0 | 1.0689 | 1.0781 | +0.9% | 0.014 | 0.51 | 96.7% |
| ja 1 | 0.9444 | 0.9596 | +1.6% | 0.047 | 1.47 | 94.0% |
| en 2 | 1.6266 | 1.5577 | −4.2% | 0.342 | 27.50 | 90.9% |
| agent 3 | 1.4287 | 1.4406 | +0.8% | 0.043 | 1.48 | 93.5% |
| code 4 | 2.0366 | 2.0579 | +1.0% | 0.042 | 0.88 | 90.5% |
| ja 5 | 1.6522 | 1.5681 | −5.1% | 0.237 | 23.10 | 89.0% |
| en 6 | 2.1901 | 2.1897 | −0.0% | 0.164 | 18.62 | 83.5% |
| agent 7 | 1.4225 | 1.3818 | −2.9% | 0.424 | 19.48 | 86.7% |
- By category (sum of 2 samples): code +1.0%, Japanese −2.7%, English −1.8%, agent transcripts −1.0%. All 8 samples are within +2% of dense.
- Mean KL: 0.141 without the agent samples, 0.164 with them.
- Q8_0, same 8 samples at 128K (dense = the Q8_0 file with
STRATA_Q35_QSA=0):- By sample: code +1.0 / +0.9%, Japanese +3.7% / −3.0%, English −5.1 / −1.5%, agent +2.5% / −1.7%.
- By category: code +0.9%, Japanese −0.7%, English −2.9%, agent −0.0%.
- Mean KL: 0.150 without the agent samples, 0.222 with them.
- Two samples are more than 2% worse than dense: Japanese 1 and agent 3. We ship it as is, as BETA, with no tuning toward these samples.
- Needle at 125K (3 secrets plus a function's value, through the server, thinking on):
- Results: 4/4 with QSA, 4/4 dense, on the same file and a 123,818-token prompt.
- Thinking took 186 tokens with QSA and 224 dense; the model closed it on its own both times.
The KL is 4x Qwen's: what that means
- With the same settings and samples, the mean KL is 0.141 here vs 0.032 for Qwen3.5-9B QSA (Q4_K_M, without the agent samples). nll did not get worse: QSA is below dense on 4 of the 8 samples.
- To see how much this model moves with no QSA at all, we compared dense Q4_K_M to dense Q8_0, both with
STRATA_Q35_QSA=0:
| Ornith-1.5-9B | Qwen3.5-9B | |
|---|---|---|
| quantization only: dense Q4_K_M vs dense Q8_0, KL (6 samples, no agent) | 0.318 | 0.072 |
| same, all 8 samples | 0.450 | 0.186 |
| QSA vs dense, Q4_K_M, KL (6 samples, no agent) | 0.141 | 0.032 |
| ratio (QSA ÷ quantization only) | 0.44 | 0.44 |
- Ornith's 128K predictions move about 4.4x as much as Qwen's from quantization alone. Relative to that, QSA moves them by the same ratio on both models, 0.44.
- In BF16 (PyTorch, no quantization) the gap is the same:
- On these 8 samples at 128K, the shipped QSA settings give KL 0.150 for Ornith and 0.037 for Qwen.
- Even an oracle that picks the truly best blocks gives KL 0.236 vs 0.076 (32K, one sample per category).
- The indexer itself catches as much attention as Qwen's: capture 0.853 vs 0.859 at 128K.
- Our reading: Ornith-1.5-9B itself is more sensitive at 128K. QSA does not do worse on it than on Qwen.
- Dense itself is unstable at some positions: every setting has positions with a max KL above 20.
Speed: QSA vs dense
The setup:
- RTX 5090 32 GB, Abyss, Q4_K_M, MTP drafts on, 256 tokens generated, greedy.
- One 129,872-token chat prompt, 2 runs each, dense and QSA interleaved.
- The two runs of each setting gave the same output (same md5).
| dense | QSA | ||
|---|---|---|---|
| decode, 128K | 138.1 / 138.0 tok/s | 214.8 / 215.0 tok/s | 1.56x |
| prompt, 128K | 6,040 / 6,027 tok/s | 5,554 / 5,559 tok/s | −7.9% |
| MTP draft acceptance | 86.6% | 82.8% |
- Qwen3.5-9B QSA, on the same PC with the same prompt, gets 1.81x (132.7 → 239.9 tok/s).
- The gain is smaller here because MTP acceptance drops with QSA on Ornith (86.6% → 82.8%). On Qwen it rose (84.1% → 92.1%).
- These numbers come from one prompt. Acceptance depends on the text, so other prompts can give other ratios.
- Prompt processing is about 8% slower at 128K. That is the indexer's own work, and Qwen showed the same.
Memory (RTX 5090, Q4_K_M, QSA on, MTP on)
| context | VRAM cap | GPU peak | host RAM peak | KV | decode |
|---|---|---|---|---|---|
| 32K | none | 11.17 GB | 6.96 GB | 1.12 GB | 312 tok/s |
| 64K | none | 12.40 GB | 6.96 GB | 2.25 GB | 280 tok/s |
| 128K | none | 14.86 GB | 6.97 GB | 4.50 GB | 224 tok/s |
| 128K | 11 GB (STRATA_GLM_VRAM_GB=11) |
not measured |
- GPU peak is the strata process's usage from nvidia-smi, read every 0.2 s. Host RAM is the process's peak RSS.
- With no cap, 4 GB of the peak is a prompt buffer.
- The uncapped peaks match Qwen3.5-9B QSA exactly.
- For Qwen, the buffer shrank under a cap: 128K ran in 10.68 GB at an 11 GB cap at the same speed. Ornith should behave the same, but we have not measured it, and we have not tested a real 12 GB card.
Japanese
These checks used Q4_K_M through the server, with temperature 0 and thinking on, for both QSA and dense on the same file.
- Three short prompts (quantum computers in 3 sentences, a Python dedup function, a delivery-delay apology email):
- Natural Japanese in all three, and QSA and dense gave the same text.
- These prompts are too short for QSA to act (under 2,051 cells), so this checks the GGUF itself.
- The email has one slightly odd phrase. That comes from the model, not from QSA.
- A 50,849-token Japanese document with two questions: QSA got 2/2 and dense got 2/2. The wording differed, but both were right.
Feedback (this is a BETA)
- Speed on your GPU: post the server log's speed lines with this issue form, with your GPU and context length.
- Quality in real use: answers that differ from dense (
STRATA_Q35_QSA=0) on the same prompt help most. - A 12 GB card or a VRAM cap: the 11 GB case above is not measured on this model yet.
Credits and license
- ornith-ai: the model (MIT). The weights here are theirs, quantized.
- higa2515: the QSA approach for Qwen-family models. The indexer's form and tensor names follow their Qwen3.8-27B QSA release.
- MIT, as the original model.
日本語
- NVIDIA の GPU 12 GB 以上向け(想定)。RTX 5090 で、128K の文脈が GPU 14.9 GB、RAM 7 GB で動きました。
- VRAM を制限した計測は、このモデルではまだしていません。
- 同じ形の Qwen3.5-9B QSA は、11 GB に制限して 10.7 GB で動きました。同じになると見ていますが、推定です。
- Abyss 専用です。 llama.cpp は試していません。同じ作りの Qwen3.5-9B QSA は
wrong number of tensorsで読めませんでした。 - ornith-ai/Ornith-1.5-9B に QSA の indexer を足したものです。
- 重みは Ornith のまま(量子化だけ)です。足したのは小さな indexer(2,100 万パラメータ)です。
- 作り方は Qwen3.5-9B QSA と同じです。学習データも同じです。
- indexer は、借りた RTX 5090 1 枚で 3 時間 45 分(2.16 kWh)かけて学習しました。
- full attention の 8 層のうち、層 3 の 1 層は dense のまま、ほかの 7 層で QSA を使います。
STRATA_Q35_QSA=0で全部 dense にできます。 - GGUF には Ornith 自身のチャットテンプレート(chat_template.jinja)を入れました。
- 元の tokenizer_config.json には Qwen3.5 のテンプレートが残っています。そのため、普通に変換すると Qwen のテンプレートが入ります。
- BETA です。 測ったのは 1 台の PC です。
| dense | QSA | |
|---|---|---|
| 生成(128K) | 138 tok/s | 215 tok/s(1.56 倍) |
| プロンプト(128K) | 6,034 tok/s | 5,556 tok/s(−7.9%) |
| MTP の採択率 | 86.6% | 82.8% |
- Qwen3.5-9B QSA は 1.81 倍でした。Ornith で倍率が低いのは、QSA で MTP の採択率が下がるためです(Qwen では上がりました)。数字は 1 つのプロンプトで測ったものです。
- 品質(128K、8 本、同じファイルの dense と比べた nll の差):
- Q4_K_M: 8 本すべて dense の +2% 以内でした。カテゴリ別ではコード +1.0%、日本語 −2.7%、英語 −1.8%、エージェント −1.0%。
- Q8_0: 日本語 1(+3.7%)とエージェント 3(+2.5%)が +2% を超えました。そのまま BETA として出しています。
- KL は Qwen の 4 倍あります(0.141、Qwen は 0.032)。
- ただし Ornith は、QSA なしでも量子化だけで KL 0.318 動きます(Qwen は 0.072)。
- QSA の影響を「量子化だけの影響」と比べた比は、両方とも 0.44 で同じです。
- BF16 でも同じ傾向でした。Ornith 自体が、128K で予測の揺れやすいモデルだと見ています。
- 125K のニードル(秘密 3 つ + 関数の値):QSA 4/4、dense 4/4。
- 日本語:
- 短い 3 問(量子コンピュータの説明、Python の関数、お詫びメール)は、自然な日本語で答えました。
- 5 万トークンの日本語の文書への問い 2 つは、QSA も dense も 2/2 でした。
- GPU メモリ: 32K / 64K / 128K で 11.2 / 12.4 / 14.9 GB(Qwen3.5-9B QSA と同じ)。
- Abyss v1.3.162 以降が必要です。それより前の版では全部 dense で動きます。
- v1.3.166 からは、まとめてセットアップで入れられます。
./setup-deepseek.sh --model ornith-9b-qsa --download-model- Windows は
setup-deepseek.batに同じ引数です。
- 元のモデルと同じ MIT ライセンスです。QSA の考え方は higa2515 さんの Qwen3.8-27B-QSA によります。
- 速さの報告は この issue フォーム にお願いします。12 GB の GPU や VRAM を制限した結果は、特に助かります。