SAGE FRED-T5 distilled 95M — Q8_0 GGUF

Q8_0 conversion of ai-forever/sage-fredt5-distilled-95m, a Russian spelling and punctuation corrector from ai-forever/SberDevices. This is an encoder–decoder model, not a chat or instruction-following LLM.

The original model and its weights belong to their authors and are released under the MIT license. This repository contains a quantized conversion, not a fine-tune.

File

SHA-256:

0dd57ccdbd1abad6b22c658a386c94668e1d241c06125973b8ad869d2f1f2a8e

Conversion

• Original source revision: ed51b4a46603931380951a3d8456685c9215864f.
• Exported through a lossless FP32 GGUF intermediate, then quantized directly to Q8_0; no lower-precision requantization.
• All 190 FP32 tensor payloads were checked against the original safetensors.
• The Q8_0 file contains 146 quantized tensors and 44 unchanged FP32 tensors.
• The model uses GPT-2 byte-level BPE despite having a T5 architecture. The conversion preserves the actual encoder BOS <LM> ID 50357, EOS 2, and decoder-start/pad ID 0.
• No imatrix was used for this Q8_0 release. No Q6/IQ4 variants are included.

Run with llama.cpp

An ordinary stock llama.cpp b10358 (030ebb558), using llama-completion, was tested successfully on CPU. The three correction examples below matched original Transformers greedy output.

Put the input text in a UTF-8 input.txt file, then run:

llama-completion \
  -m sage-fredt5-distilled-95m-Q8_0.gguf \
  -f input.txt \
  -ngl 0 -c 512 -b 512 -ub 512 -t 8 -n 256 \
  --temp 0 --no-warmup --no-display-prompt -fa off

On Windows the executable is llama-completion.exe. Use the encoder-aware completion/API path, not a chat template. No custom runtime binaries, DLLs, Python wrapper or local environment files are distributed in this repository.

Runtime compatibility caveat

Stock llama.cpp b11583 reproduced a T5 no-cache attention regression: keys/values were padded to a multiple of 256, but the relative-position bias was not padded with them. That build aborted during graph construction. This is runtime-specific, not a requirement for a special GGUF format; stock b10358 completion worked without a patch. Do not assume every chat-oriented frontend or every llama.cpp build supports this encoder–decoder path. LM Studio and GPU inference were not tested for this release.

Functional checks

These are small functional regression checks, not a full spelling-quality evaluation. The original model can make incorrect corrections, and complex formatting may be difficult. Process sentence-sized segments within the model context instead of assuming unlimited document length. Consult the original model card for training details, limitations and original benchmark results; those metrics were not re-measured for this quantization.