Loading…
Loading…
GGUF exports for kai-os/carnice-v2-27b, a merged BF16 SFT of Qwen/Qwen3.6-27B for Hermes-style agent traces.
| File | Size class | Use |
|---|---|---|
carnice-v2-27b-IQ2_M.gguf |
9.4GB | Best 16GB-GPU target. Built with a Carnice/Hermes imatrix calibration pass. |
carnice-v2-27b-Q2_K.gguf |
10GB | Safest 16GB-GPU fallback. More compatible than IQ quants, lower quality than imatrix IQ2_M. |
carnice-v2-27b-Q4_K_M.gguf |
16GB | Balanced local quality tier. May require shorter context or partial CPU offload on a 16GB GPU. |
carnice-v2-27b-Q5_K_M.gguf |
18GB | Better quality tier for 24GB+ or split/offload setups. |
carnice-v2-27b-Q8_0.gguf |
27GB | Near-lossless quant tier for high-memory systems. |
carnice-v2-27b-bf16.gguf |
51GB | Full BF16 GGUF export. |
For a 16GB GPU, start with IQ2_M if your runtime supports IQ quants and this Qwen3.5/Qwen3.6 GGUF architecture. If the runtime is older or fails to load IQ quants, use Q2_K.
| Metric | Qwen3.6-27B base | Carnice SFT |
|---|---|---|
| IFEval prompt strict, limit 20 | 85.0% | 90.0% |
| IFEval prompt loose, limit 20 | 85.0% | 90.0% |
| IFEval instruction strict, limit 20 | 90.0% | 93.3% |
| IFEval instruction loose, limit 20 | 90.0% | 93.3% |
| Held-out assistant-token eval loss | 0.607 | 0.414 |
| Held-out assistant-token eval perplexity | 1.835 | 1.513 |
Scope note: these are source SFT checks, not separate GGUF quant benchmark scores. The full benchmark artifact bundle is in the merged model repo: kai-os/carnice-v2-27b.
This model converts as qwen35 GGUF with hybrid attention/SSM layers. Use a recent llama.cpp build; older GGUF runtimes may not know this architecture yet.
Example:
llama-cli \
-m carnice-v2-27b-Q2_K.gguf \
-ngl all \
-c 8192 \
-p "Write a short plan for a Hermes agent debugging a failing tool call."
For long context on 16GB, keep the weight quant low and tune KV cache aggressively. The file fitting in VRAM does not mean 128K context will also fit.
kai-os/Carnice-V2-27b