Meno-Lite-0.1 GGUF
This repository contains quantized GGUF versions of Meno-Lite-0.1.
All variants were produced using an importance matrix computed on the train split of the ru_llm_calibration dataset, and are intended to be run with llama.cpp.
Available Formats
| Quantization type | File size | Quality | Recommendation |
|---|---|---|---|
| Q8_0 | ~8.05 GB | Virtually identical to FP16 | Best quality. Ideal for CPU inference when memory is not a constraint. |
| Q5_K_M | ~5.41 GB | Minimal degradation | Recommended balance. Excellent speed and quality, fits most consumer GPUs. |
| Q4_K_M | ~4.65 GB | Moderate degradation | "Golden standard". Best trade-off between size and quality. |
| IQ3_M | ~3.54 GB | Noticeable degradation | Maximum memory savings. Quality drops visibly; suited for highly constrained devices. |
Quality Evaluation
Quality was measured on the test split of the Ru LLM Calibration dataset using the llama-perplexity utility. The original FP16 model served as the reference.
| Metric | Q8_0 | Q5_K_M | Q4_K_M | IQ3_M |
|---|---|---|---|---|
| Mean PPL (Q) ↓ | 9.047 | 9.075 | 9.135 | 9.689 |
| PPL correlation ↑ | 99.97% | 99.87% | 99.69% | 98.64% |
| Mean KLD ↓ | 0.0020 | 0.0077 | 0.0174 | 0.0804 |
| Same top p ↑ | 96.71% | 94.36% | 92.16% | 84.58% |
↑ – higher is better; ↓ – lower is better
How to interpret these metrics:
- Mean PPL (Q): Lower is better. Shows the average perplexity of the quantized model.
- PPL correlation: Closer to 100% indicates the quantized model behaves almost identically to FP16. Values above 99.5% are considered excellent.
- Mean KLD: Measures the divergence between the output probability distributions. Lower is better; 0 means identical distributions.
- Same top p: The percentage of tokens where the quantized model's top prediction matches the FP16 model. Higher is better – it reflects how often the model's first-choice token remains unchanged.
Usage
1. Install llama.cpp
Follow the official build instructions.
2. Run the model
# CLI
./llama-cli -hf bond005/meno-lite-0.1-gguf -m meno-lite-0.1-Q4_K_M.gguf -p "Привет, как дела?"
# Server with WebUI (default http://127.0.0.1:8080)
./llama-server -hf bond005/meno-lite-0.1-gguf -m meno-lite-0.1-Q4_K_M.gguf --host 0.0.0.0 --port 8080
For more details on available parameters, see the llama.cpp documentation.
About Meno-Lite-0.1
Meno-Lite-0.1 is a 7B model based on Qwen2.5, fine-tuned for RAG, document QA, information extraction, and knowledge graph construction. Read more about its capabilities, training procedure, and limitations in the main model card.
License
All quantized variants inherit the license of the original model (Apache 2.0).