Bayon Instruct — GGUF quants

This repository contains GGUF conversions and quantizations of attentionlab/bayon-it (Bayon Instruct), a ~109M-parameter Gemma-architecture decoder-only language model for Khmer.

The original FP32 weights live in the upstream repo. To re-quantize yourself, start from the original safetensors files:

• Repo: attentionlab/bayon-it
• Weights: model.safetensors
• Tokenizer: tokenizer.model · tokenizer.vocab

Bayon Instruct is instruction-tuned (LoRA) from attentionlab/bayon, trained with data from attentionlab/bayon-sft. Full details (intended use, evaluation, training recipe, limitations) are in the original model card.

Files

All three files come from a single BF16 conversion; the quants are mixed per-tensor types (57 f32 norm tensors are preserved), matching what llama-quantize produces by default.

Usage

Requires a reasonably recent llama.cpp (GGUF V3, gemma architecture; build used: 0.5.0-dev, 2026-10-17).

# Quick test
llama-cli -m attentionlab_bayon-it-q8_0.gguf -p "សួស្តី ពិភពលោក"

# Serve an OpenAI-compatible endpoint
llama-server -m attentionlab_bayon-it-q4_k_m.gguf -c 2048

More general quantization-format tips: llama.cpp/gguf docs.

How these were produced

Converted with llama.cpp (convert_hf_to_gguf.py + llama-quantize, build 0.5.0-dev / commit unknown, built 2026-10-17 via the llama-cpp-python 0.3.36 vendored sources):

python3 convert_hf_to_gguf.py models/attentionlab/bayon-it --outtype bf16 \
        --outfile output/attentionlab_bayon-it-bf16.gguf

llama-quantize output/attentionlab_bayon-it-bf16.gguf output/attentionlab_bayon-it-q8_0.gguf q8_0
llama-quantize output/attentionlab_bayon-it-bf16.gguf output/attentionlab_bayon-it-q4_k_m.gguf q4_k_m

Original model (summary from upstream)

See the original model card for intended use, benchmark results (Khmer history & culture, language-bleed ratio), training details, limitations, and safety considerations.