Lythri-7B-A4B-GGUF

This repository contains GGUF quantizations of Lythri-7B-A4B for use with llama.cpp and compatible apps. For model details, evaluation results and limitations, see the original model card.

Note: Lythri models are highly sensitive to quantization. We recommend using the highest-precision quantization your hardware allows. Lower-bit quantizations still work but are not recommended.

Usage

llama.cpp

llama-cli -hf Lythri/Lythri-7B-A4B-GGUF --temp 0.7 --top-p 0.9 --top-k 64 --repeat-penalty 1.05

Ollama

ollama run hf.co/Lythri/Lythri-7B-A4B-GGUF

LM Studio

Search for Lythri/Lythri-7B-A4B-GGUF in the model browser.

To use a specific quantization, append its type to the repository name, for example Lythri/Lythri-7B-A4B-GGUF:Q8_0.

License

Lythri is built on Gemma 4 and is released under the Apache License 2.0.