Kev 4B GGUF (ggmlc)
English System 1 decision model compiled from jaredpalmer/kev-4b (Qwen3.5-4B hybrid Gated DeltaNet, LoRA merged in fp32 before export, context 2048).
These files are not llama.cpp / llama-cli GGUFs. They are produced by ggmlc. Loading them in llama.cpp will fail.
Typed questions (choice / score / noul) are scored in one forward. There is no autoregressive token generation. Tokenizer, sequence program, and postprocess live in the GGUF (ggmlc.decision).
Source and CLI: examples/laya
Smaller siblings: kev-0.5b-GGUF, kev-0.8b-GGUF
Files
| File | Quant | Size | Notes |
|---|---|---|---|
kev_4b_f16.gguf |
F16 | ~8.05 GB | Best numerical match. Qwen3.5 zero-centered RMS is fused. |
kev_4b_q8_0.gguf |
Q8_0 | ~4.29 GB | Uniform Q8_0. 1D norms stay F32. |
kev_4b_ud_q4_k_m.gguf |
UD_Q4_K_M | ~4.02 GB | Embedding stays F16, sensitive projections Q8_0, other weights Q4_0. Smaller than uniform Q8_0. |
huggingface-cli download mys/kev-4b-GGUF kev_4b_q8_0.gguf --local-dir .
Run with laya
laya info kev_4b_q8_0.gguf
laya decide kev_4b_q8_0.gguf --preset email --device auto --cuda-graph
laya serve kev_4b_q8_0.gguf --port 8080 --device auto --cuda-graph
laya bench kev_4b_q8_0.gguf --preset email --device auto --cuda-graph
serve is Decision Studio plus POST /v1/systemone (TypeSafe). The advertised model id is kev-4b.
License
Apache 2.0, same as jaredpalmer/kev-4b. Compiler: ggmlc (MIT).