🐦 Colibri — GGUF (llama.cpp)

GGUF builds of tardellirs/colibri-embed-ptbr, a ~157M-parameter Brazilian-Portuguese embedding model, for llama.cpp (llama-server, llama-embedding) and the runtimes built on it.

The two Dense projection layers of the SentenceTransformer pipeline are inside the file, and the tokenizer is the model's own 64k BPE vocabulary, so the token ids are exactly those of the HF model. Every file gives the same 768-dim embeddings as the original, up to quantization.

Q8_0 is the one to use: indistinguishable from the original at about a quarter of the PyTorch RAM. Q6_K and Q5_K_M are still close; Q4_K_M is the smallest file but loses the most, and uses more RAM than Q5_K_M.

Measured on a Verda CPU.4V.16G (4 vCPU AMD EPYC, 16 GB), llama.cpp a4d4f2f (2026-11-09), llama-server with 4 threads. Peak RAM is the server process's peak resident memory. Latency is one sentence per request; throughput is 512 sentences sent 32 per request. Cosine is against the HF model in fp32 on 30 Brazilian-Portuguese sentences.

Usage

llama-server -m colibri-embed-ptbr-Q8_0.gguf --embeddings --pooling mean -c 2048 -ub 2048 -b 2048

import requests

def embed(texts):
    r = requests.post("http://127.0.0.1:8080/v1/embeddings", json={"input": texts})
    return [d["embedding"] for d in r.json()["data"]]

# llama.cpp doesn't add the EmbeddingGemma prompts: put them in yourself
q = embed(["task: search result | query: Como declarar imposto de renda de aluguel?"])
d = embed(["title: none | text: Rendimentos de aluguel devem ser informados na ficha de Rendimentos Tributáveis..."])

Other prompts (classification, clustering, similarity, …) are listed in the original model's config_sentence_transformers.json. Embeddings come back L2-normalized, so the dot product is the cosine.

How these were made

From the original model with two fixes to its tokenizer (now also in the original repo), then convert_hf_to_gguf.py --sentence-transformers-dense-modules and llama-quantize:

1. The trim left 6,246 unused added tokens (<unused0…>, <image_soft_token>, …) with ids past the 64,000-row embedding matrix. They were removed; no text tokenizes differently.
2. The trimmed vocabulary changes the pre-tokenizer hash llama.cpp uses to recognize the tokenizer, so the hash was registered as Gemma's pre-tokenizer, which is unchanged (embedding-vocab-trimmer, gguf_bpe_patch.py).