Qwythos-9B-v2-PerfectSplit-GGUF
A mixed-precision GGUF quantization of Qwythos-9B-v2 using a "Perfect Split" strategy. The model's 427 tensors are selectively quantized: critical reasoning pathways (attention, DeltaNet/SSM, embeddings, output head) stay at full precision, while the redundant MLP layers are compressed.
🧬 Architecture Context
Qwythos-9B-v2 is a hybrid architecture combining:
- Standard attention layers (Q/K/V/O projections with QK-norm)
- DeltaNet/SSM recurrence layers (alpha, beta, dt, conv1d, ssm_out gates)
- Standard MLP blocks (gate, up, down projections with 3x expansion ratio)
This hybrid design means different tensor classes have different sensitivity to quantization:
- Attention & SSM gates control information routing and long-range memory. Errors here compound across the 1M-token context window.
- MLP layers contain massive mathematical redundancy due to the 3x expansion ratio. They tolerate aggressive quantization with minimal quality impact.
🎯 The "Perfect Split" Strategy
| Tensor Class | Precision | Count | Rationale |
|---|---|---|---|
Token Embeddings (token_embd.weight) |
F16 | 1 | First point of contact with input; errors propagate everywhere |
Output Head (output.weight) |
F16 | 1 | Maps hidden states → vocabulary logits; critical for token selection |
Output Norm (output_norm.weight) |
F32 | 1 | Final normalization before logits |
Attention Projections (attn_q, attn_k, attn_v, attn_output, attn_qkv, attn_gate) |
F16 | ~96 | Prevents attention score drift at long context |
Attention Norms (attn_norm, attn_q_norm, attn_k_norm, post_attention_norm) |
F32 | ~64 | Activation scaling stability |
DeltaNet/SSM (ssm_alpha, ssm_beta, ssm_dt, ssm_conv1d, ssm_out, ssm_a) |
F16 | ~96 | The recurrence S_t = α·S_{t-1} + β·(k⊗v) runs millions of steps; errors accumulate multiplicatively |
SSM Norms (ssm_norm) |
F32 | ~32 | Recurrence stability |
MLP Layers (ffn_gate, ffn_up, ffn_down) |
Q5_K_M | 96 | Compressed — high redundancy, low sensitivity |
Final Statistics
| Metric | Value |
|---|---|
| Original Size (BF16) | 16.69 GB |
| Final Size | 10.78 GB |
| Compression | 35.4% smaller |
| Average BPW | 10.34 |
| F16 Tensors | 331 |
| Q5_K_M Tensors | 96 |
| Total Tensors | 427 |
| Layers | 32 (0 to 31) — all intact |
Per-Tensor Compression Example (from quantization log)
blk.29.attn_q.weight : bf16 → f16 64.00 MiB → 64.00 MiB (preserved) blk.29.ssm_out.weight : bf16 → f16 32.00 MiB → 32.00 MiB (preserved) blk.29.ffn_gate.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%) blk.29.ffn_up.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%) blk.29.ffn_down.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%)
📊 Expected Quality Profile
Because the quantization is surgical (only compressing MLP layers), the quality profile differs from uniform quants:
| Capability | Expected Quality | Why |
|---|---|---|
| Chain-of-Thought reasoning | Excellent | Attention + SSM pathways at F16 preserve reasoning flow |
| Long-context recall (128K+) | Excellent | DeltaNet recurrence at F16 prevents drift |
| Code generation | Very Good | MLP compression has minimal impact on coding patterns |
| Factual knowledge recall | Very Good | Slight loss on obscure facts (MLP stores knowledge) |
| Tool-call / JSON output | Very Good | Output head at F16 preserves format fidelity |
| Overall perplexity | ~0.05 above BF16 | vs. ~0.18 above BF16 for uniform Q5_K_M |
Trade-off: This model is ~4 GB larger than a uniform Q5_K_M (~6.5 GB), but delivers noticeably better quality on reasoning-heavy tasks.
🚀 Usage
With llama.cpp
./llama-cli \
-m Qwythos-9B-v2-PerfectSplit.gguf \
--prompt "You are an expert Python developer. Solve this step by step inside tags.\n\nTask: Implement a thread-safe sliding-window rate limiter." \
--n-predict 2000 \
--temp 0.6 \
--top-k 20 \
--top-p 0.95 \
--min-p 0.05 \
--repeat-penalty 1.05 \
--ctx-size 8192 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--flash-attn
With LM Studio
Import the GGUF file
Recommended settings:
Temperature: 0.6
Top-K: 20
Top-P: 0.95
Min-P: 0.05
Repeat Penalty: 1.05
Context Size: 8192 (or higher if RAM allows)
KV Cache Type: q8_0 (important — keeps total VRAM usage reasonable)
With Ollama
Create a Modelfile:
FROM ./Qwythos-9B-v2-PerfectSplit.gguf
PARAMETER temperature 0.6
PARAMETER top_k 20
PARAMETER top_p 0.95
PARAMETER min_p 0.05
PARAMETER repeat_penalty 1.05
PARAMETER num_ctx 8192
ollama create qwythos-perfect -f Modelfile
ollama run qwythos-perfect