Aronstk15/DeepHat-V1-7B-Mixed-IQ4
This is a custom mixed-precision quantization of DeepHat/DeepHat-V1-7B.
🧠 Quantization Details
- Base Model: DeepHat-V1-7B
- Attention & LM Head Layers: Kept at FP16 to preserve reasoning, routing, and coding accuracy.
- Feed-Forward (FFN) Layers: Quantized to IQ4_XS (4-bit non-linear) using an importance matrix (
imatrix.dat) calibrated on code and technical text. - Tool Used:
llama.cpp(llama-imatrixandllama-quantize)
💡 Why Mixed Precision?
By keeping the highly sensitive Attention and Output layers in FP16, this model retains nearly 100% of the original model's intelligence while significantly reducing the file size and VRAM footprint.
🚀 How to use
You can run this model using llama.cpp, Colibrí, or any GGUF-compatible inference engine.
# Example using llama.cpp / Colibrí
./llama-server -m DeepHat-Mixed-IQ4.gguf -ngl 99 -c 8192