Boris-125M-Instruct-GGUF

GGUF quantizations of KSP-NMAI/Boris-125M-Instruct for llama.cpp and compatible runtimes (llama-server, LM Studio, koboldcpp, Jan).

The original safetensors weights live in the base repo — use those for finetuning or for any PyTorch-based runtime. GGUF is inference-only.

Which file should I pick?

Use Q8_0, or F16 if you want the exact reference weights.

⚠️ Not recommended: IQ1_S. Included for completeness only.

TQ1_0, TQ2_0 and Q1_0 were built and tested but produced unusable output at this scale, so they are deliberately not published here.

Boris-125M is a small model, and quantization behaves differently at this scale than it does for 7B+ models. The token embedding table is a large fraction of the parameters and is kept at high precision by llama.cpp, which sets a hard floor on file size. The practical result: every file here is between 49 MB and 240 MB. Dropping from Q8_0 to IQ1_S saves you a few tens of megabytes while degrading output substantially. The aggressive quants are provided for completeness, not because they are a good trade.

Files

All quantizations below 8-bit were produced with an importance matrix calibrated on 100 chunks of held-out data drawn from the model's own training mixture (60% fineweb-edu / 40% dclm).

Usage

# straight from the Hub
llama-server -hf KSP-NMAI/Boris-125M-Instruct-GGUF:Q8_0 --jinja

# or a local file
llama-server -m Boris-125M-Instruct-Q8_0.gguf --jinja

The Alpaca chat template is embedded in every file, so --jinja applies the correct prompt format automatically.

Prompt format

### Instruction:
{your instruction}

### Response:

Limitations

This is a very small instruction-tuned model. It will produce text that is frequently inaccurate, inconsistent, or offensive, and has received no alignment or safety tuning beyond supervised fine-tuning on Alpaca. Do not rely on it for factual information or deploy it without supervision.

License

Apache 2.0. Copyright 2026 OpenCerebral. See the base repository for the full notice.