Sign up

wepiqx/Ornith-1.0-9B-ASHQ1-MTP-GGUF

🤗 On Hugging Facetext-generationmit25 GBGGUF✓ HF checksums availableupdated today
Magnet

Ornith-1.0-9B — ASHQ1 Quantization (MTP-Aware)

ASHQ1 quantization of Ornith-1.0-9B with explicit MTP (Multi-Token Prediction) handling — MTP heads are kept at Q8_0 and excluded from the classifier budget, preserving their quality while the priority queue optimises the main transformer layers.

Huge thanks to protoLabsAI/Ornith-1.0-9B-MTP-GGUF for providing the BF16 source with MTP heads — we adapted it with ASHQ1 to deliver both speed and quality.

Note: File names contain "BF16" for HuggingFace parser compatibility — these are ASHQ1 quants, not BF16.

Quants

| File | Size | Method | Description |

|:-----|:----:|:------:|:------------|

| Ornith-1.0-9B-MTP-BF16-ASHQ1-4850.gguf | 4.7 GB | ASHQ1 (v6, --allow-q3-or-lower) | Tiny — uses IQ tiers to fit 4.7 GB, MTP heads at Q8_0 |

| Ornith-1.0-9B-MTP-BF16-ASHQ1-5500.gguf | 5.4 GB | ASHQ1 (v6) | Compact — beats i1-Q6_K (7.0 GB), 23% smaller |

| Ornith-1.0-9B-MTP-BF16-ASHQ1-6500.gguf | 6.4 GB | ASHQ1 (v6) | Balanced — near i1-Q8_0 quality at 33% smaller |

| Ornith-1.0-9B-MTP-BF16-ASHQ1-7300.gguf | 7.2 GB | ASHQ1 (v6) | Best — surpasses i1-Q8_0 quality, 25% smaller |

All comparisons are against i1 quants (uniform quantization with bartowski's Ornith imatrix). ASHQ1 consistently beats them at smaller sizes: the priority queue allocates bits where they matter most. See wepiqx/ASHQ1 for full benchmarks across architectures. MTP heads are deployed at Q8_0 and excluded from the classifier budget.

Note: The 4850 variant uses IQ tiers via --allow-q3-or-lower to fit the tight budget — quality is roughly on par with Q4_K_S, while 5500+ delivers a significant step up. It's best suited for lighter tasks, while coding and agentic workloads may prefer 5500+. Feel free to try it either way.

How ASHQ1 Works

See wepiqx/ASHQ1 for the full method, source code, and benchmarks.

Usage

llama.cpp

# Chat mode with built-in Jinja template (recommended)
llama-cli \
  -m Ornith-1.0-9B-MTP-BF16-ASHQ1-6500.gguf \
  --jinja \
  -ngl 99 \
  -c 8192

Recommended sampling: temperature 0.6–1.0, top_p 0.95, top_k 20.

Ollama

The model's built-in Jinja template is auto-detected by Ollama — no need to specify TEMPLATE manually.

1. Create a Modelfile:

FROM ./Ornith-1.0-9B-MTP-BF16-ASHQ1-6500.gguf

PARAMETER num_ctx 8192
PARAMETER temperature 0.6
PARAMETER top_k 20
PARAMETER top_p 0.95

2. Build and run:

ollama create ornith-ashq1-mtp-6500 -f Modelfile
ollama run ornith-ashq1-mtp-6500

LM Studio

1. Open LM Studio

2. Drag the desired GGUF file (4850, 5500, 6500, or 7300) into the app

3. Set GPU Offload to 99 layers

4. Set context length to 8192+

5. Set preset sampling: temperature 0.6, top_p 0.95, top_k 20

6. Start chatting

Quantization Configs

Generated by ASHQ1 priority queue (v6) — files in this repo were produced with these exact llama-quantize arguments.

4850 Config (--allow-q3-or-lower)

--output-tensor-type Q5_K
--token-embedding-type Q5_K
--tensor-type "(blk|BLK)\.(32)\.nextn=Q8_0"
--tensor-type "(blk|BLK)\.(32)\.ffn_down=Q4_K"
--tensor-type "(blk|BLK)\.(31)\.ffn_down=IQ4_XS"
--tensor-type "(blk|BLK)\.(30)\.ffn_down=IQ3_S"
--tensor-type "(blk|BLK)\.(32)\.attn_v=Q4_K"
--tensor-type "(blk|BLK)\.(32)\.attn_q=Q4_K"
--tensor-type "(blk|BLK)\.(32)\.attn_k=Q4_K"
--tensor-type "(blk|BLK)\.((?:26|27|28|29))\.ffn_down=IQ3_XXS"
--tensor-type "(blk|BLK)\.((?:22|23|24|25))\.ffn_down=IQ2_S"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21))\.ffn_down=IQ2_XXS"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_k_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_a=F16"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.attn_norm=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_q_norm=F16"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.post_attention_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_conv1d=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_dt=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|31)\.attn_q=Q5_K"
--tensor-type "(blk|BLK)\.(0|6|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.attn_qkv=Q5_K"
--tensor-type "(blk|BLK)\.(0|6|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.attn_gate=Q5_K"
--tensor-type "(blk|BLK)\.(0|6|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_alpha=Q5_K"
--tensor-type "(blk|BLK)\.(0|6|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_beta=Q5_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|31)\.attn_k=Q5_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|31)\.attn_v=Q5_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_output=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.ffn_gate=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.ffn_up=Q4_K"
--tensor-type "(blk|BLK)\.([1-2]|[4-5])\.attn_gate=Q4_K"
--tensor-type "(blk|BLK)\.([1-2]|[4-5])\.ssm_beta=Q4_K"
--tensor-type "(blk|BLK)\.([1-2]|[4-5])\.ssm_alpha=Q4_K"
--tensor-type "(blk|BLK)\.([1-2]|[4-5])\.attn_qkv=Q4_K"
--tensor-type "(blk|BLK)\.(28|30)\.ssm_out=IQ4_XS"
--tensor-type "(blk|BLK)\.((?:24|25)|29)\.ssm_out=IQ3_S"
--tensor-type "(blk|BLK)\.((?:21|22)|26)\.ssm_out=IQ3_XXS"
--tensor-type "(blk|BLK)\.(18|20)\.ssm_out=IQ2_S"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17))\.ssm_out=IQ2_XXS"
--tensor-type ".*output_norm.*=F16"

5500 Config (base=Q5_K_M)

--output-tensor-type Q5_K
--token-embedding-type Q5_K
--tensor-type "(blk|BLK)\.(32)\.nextn=Q8_0"
--tensor-type "(blk|BLK)\.(3)\.attn_q=Q6_K"
--tensor-type "(blk|BLK)\.(3)\.attn_k=Q6_K"
--tensor-type "(blk|BLK)\.(31)\.ffn_down=Q5_K"
--tensor-type "(blk|BLK)\.((?:20|21)|(?:24|25)|28|30)\.attn_qkv=Q6_K"
--tensor-type "(blk|BLK)\.(3|32)\.attn_v=Q6_K"
--tensor-type "(blk|BLK)\.((?:20|21)|(?:24|25)|28|30)\.ssm_beta=Q6_K"
--tensor-type "(blk|BLK)\.((?:20|21)|(?:24|25)|28|30)\.ssm_alpha=Q6_K"
--tensor-type "(blk|BLK)\.((?:20|21)|(?:24|25)|28|30)\.attn_gate=Q6_K"
--tensor-type "(blk|BLK)\.(31)\.attn_output=Q5_K"
--tensor-type "(blk|BLK)\.(30)\.ssm_out=Q5_K"
--tensor-type "(blk|BLK)\.(1)\.attn_gate=Q4_K"
--tensor-type "(blk|BLK)\.(1)\.ssm_alpha=Q4_K"
--tensor-type "(blk|BLK)\.(32)\.attn_q=Q4_K"
--tensor-type "(blk|BLK)\.(1)\.ssm_beta=Q4_K"
--tensor-type "(blk|BLK)\.(1)\.attn_qkv=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30)|32)\.ffn_down=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_conv1d=F16"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.post_attention_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_norm=F16"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.attn_norm=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_k_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_dt=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_a=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_q_norm=F16"
--tensor-type "(blk|BLK)\.((?:26|27|28|29|30|31))\.ffn_up=Q5_K"
--tensor-type "(blk|BLK)\.(0|2|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|22|26|29)\.attn_qkv=Q5_K"
--tensor-type "(blk|BLK)\.(0|2|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|22|26|29)\.ssm_alpha=Q5_K"
--tensor-type "(blk|BLK)\.(7|11|15|19|23|27|(?:31|32))\.attn_k=Q5_K"
--tensor-type "(blk|BLK)\.(7|11|15|19|23|27|31)\.attn_v=Q5_K"
--tensor-type "(blk|BLK)\.(0|2|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|22|26|29)\.attn_gate=Q5_K"
--tensor-type "(blk|BLK)\.(7|11|15|19|23|27|31)\.attn_q=Q5_K"
--tensor-type "(blk|BLK)\.(0|2|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|22|26|29)\.ssm_beta=Q5_K"
--tensor-type "(blk|BLK)\.((?:26|27|28|29|30|31))\.ffn_gate=Q5_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25)|32)\.ffn_up=Q4_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25)|32)\.ffn_gate=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29))\.ssm_out=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|32)\.attn_output=Q4_K"
--tensor-type ".*output_norm.*=F16"

6500 Config (base=Q5_K_M)

--output-tensor-type Q5_K
--token-embedding-type Q5_K
--tensor-type "(blk|BLK)\.(32)\.nextn=Q8_0"
--tensor-type "(blk|BLK)\.(31)\.ffn_down=Q6_K"
--tensor-type "(blk|BLK)\.(0|(?:9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_beta=Q8_0"
--tensor-type "(blk|BLK)\.(0|(?:9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.attn_gate=Q8_0"
--tensor-type "(blk|BLK)\.(3|7|15|19|23|27|31)\.attn_v=Q8_0"
--tensor-type "(blk|BLK)\.(0|(?:9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.attn_qkv=Q8_0"
--tensor-type "(blk|BLK)\.(0|(?:9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_alpha=Q8_0"
--tensor-type "(blk|BLK)\.(3|7|15|19|23|27|31)\.attn_k=Q8_0"
--tensor-type "(blk|BLK)\.(3|7|15|19|23|27|31)\.attn_q=Q8_0"
--tensor-type "(blk|BLK)\.(11)\.attn_q=Q6_K"
--tensor-type "(blk|BLK)\.(11)\.attn_k=Q6_K"
--tensor-type "(blk|BLK)\.(11)\.attn_v=Q6_K"
--tensor-type "(blk|BLK)\.([1-2]|[4-6]|8)\.ssm_beta=Q6_K"
--tensor-type "(blk|BLK)\.([1-2]|[4-6]|8)\.attn_qkv=Q6_K"
--tensor-type "(blk|BLK)\.((?:14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))\.ffn_gate=Q6_K"
--tensor-type "(blk|BLK)\.([1-2]|[4-6]|8)\.ssm_alpha=Q6_K"
--tensor-type "(blk|BLK)\.((?:28|29|30))\.ssm_out=Q6_K"
--tensor-type "(blk|BLK)\.((?:14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))\.ffn_up=Q6_K"
--tensor-type "(blk|BLK)\.([1-2]|[4-6]|8)\.attn_gate=Q6_K"
--tensor-type "(blk|BLK)\.(27|31)\.attn_output=Q6_K"
--tensor-type "(blk|BLK)\.(32)\.attn_v=Q5_K"
--tensor-type "(blk|BLK)\.(32)\.attn_k=Q4_K"
--tensor-type "(blk|BLK)\.(32)\.attn_q=Q4_K"
--tensor-type "(blk|BLK)\.((?:26|27|28|29|30))\.ffn_down=Q5_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25)|32)\.ffn_down=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_q_norm=F16"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.post_attention_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_norm=F16"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.attn_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_a=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_k_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_conv1d=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_dt=F16"
--tensor-type "(blk|BLK)\.(18|(?:20|21|22)|(?:24|25|26))\.ssm_out=Q5_K"
--tensor-type "(blk|BLK)\.((?:4|5|6|7|8|9|10|11|12|13))\.ffn_gate=Q5_K"
--tensor-type "(blk|BLK)\.((?:4|5|6|7|8|9|10|11|12|13))\.ffn_up=Q5_K"
--tensor-type "(blk|BLK)\.(19|23)\.attn_output=Q5_K"
--tensor-type "(blk|BLK)\.([0-3]|32)\.ffn_up=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17))\.ssm_out=Q4_K"
--tensor-type "(blk|BLK)\.([0-3]|32)\.ffn_gate=Q4_K"
--tensor-type "(blk|BLK)\.(3|7|11|15|32)\.attn_output=Q4_K"
--tensor-type ".*output_norm.*=F16"

7300 Config (base=Q5_K_M)

--output-tensor-type Q5_K
--token-embedding-type Q5_K
--tensor-type "(blk|BLK)\.(32)\.nextn=Q8_0"
--tensor-type "(blk|BLK)\.(31)\.ffn_down=Q8_0"
--tensor-type "(blk|BLK)\.(27|31)\.attn_output=Q8_0"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|31)\.attn_v=Q8_0"
--tensor-type "(blk|BLK)\.(28|30)\.ssm_out=Q8_0"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.attn_qkv=Q8_0"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.attn_gate=Q8_0"
--tensor-type "(blk|BLK)\.((?:16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))\.ffn_up=Q8_0"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_beta=Q8_0"
--tensor-type "(blk|BLK)\.((?:16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))\.ffn_gate=Q8_0"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_alpha=Q8_0"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|31)\.attn_q=Q8_0"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|31)\.attn_k=Q8_0"
--tensor-type "(blk|BLK)\.(32)\.attn_k=Q6_K"
--tensor-type "(blk|BLK)\.((?:22|23|24|25|26|27|28|29|30))\.ffn_down=Q6_K"
--tensor-type "(blk|BLK)\.((?:4|5|6|7|8|9|10|11|12|13|14|15))\.ffn_up=Q6_K"
--tensor-type "(blk|BLK)\.((?:4|5|6|7|8|9|10|11|12|13|14|15))\.ffn_gate=Q6_K"
--tensor-type "(blk|BLK)\.(18|(?:20|21|22)|(?:24|25|26)|29)\.ssm_out=Q6_K"
--tensor-type "(blk|BLK)\.(19|23)\.attn_output=Q6_K"
--tensor-type "(blk|BLK)\.(32)\.attn_v=Q5_K"
--tensor-type "(blk|BLK)\.(32)\.ffn_gate=Q4_K"
--tensor-type "(blk|BLK)\.(32)\.ffn_up=Q4_K"
--tensor-type "(blk|BLK)\.(32)\.attn_q=Q4_K"
--tensor-type "(blk|BLK)\.((?:18|19|20|21))\.ffn_down=Q5_K"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17)|32)\.ffn_down=Q4_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_dt=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_a=F16"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.attn_norm=F16"
--tensor-type "(blk|BLK)\.((?:0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32))\.post_attention_norm=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_k_norm=F16"
--tensor-type "(blk|BLK)\.(3|7|11|15|19|23|27|(?:31|32))\.attn_q_norm=F16"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|(?:8|9|10)|(?:12|13|14)|(?:16|17|18)|(?:20|21|22)|(?:24|25|26)|(?:28|29|30))\.ssm_conv1d=F16"
--tensor-type "(blk|BLK)\.([0-3])\.ffn_gate=Q5_K"
--tensor-type "(blk|BLK)\.([0-3])\.ffn_up=Q5_K"
--tensor-type "(blk|BLK)\.((?:9|10)|(?:12|13|14)|(?:16|17))\.ssm_out=Q5_K"
--tensor-type "(blk|BLK)\.(7|11|15)\.attn_output=Q5_K"
--tensor-type "(blk|BLK)\.([0-2]|[4-6]|8)\.ssm_out=Q4_K"
--tensor-type "(blk|BLK)\.(3|32)\.attn_output=Q4_K"
--tensor-type ".*output_norm.*=F16"

Hardware Requirements

| Variant | Size (GGUF) | VRAM (ctx 8192) | Minimum GPU |

|:--------|:-----------:|:----------------:|:-----------:|

| 4850 | 4.7 GB | ~5.8 GB | 8 GB |

| 5500 | 5.4 GB | ~6.5 GB | 8 GB |

| 6500 | 6.4 GB | ~7.5 GB | 8 GB |

| 7300 | 7.2 GB | ~8.3 GB | 12 GB |

Notes