bekko-static-v1-slim-32k

A vocabulary-pruned derivative of hotchpotch/bekko-static-embedding-v1, a static embedding model (token lookup + mean pooling, 1024-d, Matryoshka 32–1024, no transformer). Same architecture, same embedding space, but 32,000 of the original 200,000 WordPiece tokens — the ones most useful for the 14 MNanoBEIR languages:

English, Arabic, German, Spanish, French, Italian, Japanese, Korean, Norwegian, Portuguese, Serbian, Swedish, Thai, Vietnamese.

Text in other languages is not supported: their script-specific tokens were removed and will tokenize to characters or [UNK].

Usage

sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("flower-computer/bekko-static-v1-slim-32k")           # add truncate_dim=256 etc. for Matryoshka sizes
emb = model.encode(["What is the capital of France?", "Paris is the capital of France."])
print(model.similarity(emb[0:1], emb[1:2]))

Queries and documents use no prompts/prefixes, as in the source model.

Rust: encoders

encoders (github.com/flowercomputers/encoders) compiles a static embedding model into your binary at build time — no runtime download, no Python. Select this model with its cargo feature:

$ cargo add encoders --features='bekko-static-v1-slim-32k'

let one   = encoders::encode_single("What is the capital of France?");
let batch = encoders::encode(["Paris is the capital of France.", "Tokyo is the capital of Japan."]);

Optional features: mrl-512 … mrl-32 truncate to a Matryoshka width (native 1024 otherwise); int8 / int16 (default) / float32 pick the table type — int16 is practically indistinguishable from f32 at half the size, int8 a quarter; rayon parallelises batches. Output matches sentence-transformers' StaticEmbedding (same tokenization, mean pooling); the crate's tests check its embeddings against the reference implementation.

The family

All members: huggingface.co/collections/flower-computer/bekko-static-v1-slim

Vocabulary size × embedding width

NanoBEIR-en — rows: vocabulary, columns: dims

MNanoBEIR — rows: vocabulary, columns: dims

NanoMIRACL — rows: vocabulary, columns: dims

The 50k+ members keep 25 languages; the 32k member keeps the 14 MNanoBEIR languages (en ar de es fr it ja ko no pt sr sv th vi), because 25 languages do not fit in 32k pieces without the long-word scripts (te/th/bn) shattering into characters. Scores are nDCG@10 × 100 at 1024 dims; NanoMIRACL includes languages the 32k model drops.

How the vocabulary was chosen

The source model's quality lives in its 200k-token table, and in a static model the tokenizer's pieces are exactly the vectors that get averaged, so removing a token is not free: every word that used it is re-tokenized into shorter pieces whose vectors were trained for other contexts. Tokens were therefore ordered by expected damage from removal:

usefulness(t) = corpus frequency of t × ‖ E[t] − Σ E[pieces t falls back to when t is removed] ‖

computed in rounds (drop the least useful 10 %, re-tokenize, re-score the tokens whose fallback changed), then truncated at the target size. Frequencies are a per-language mix of WebFAQ-v2 and Wikipedia text (equal weight per language, English ×3). Special tokens and every single-character piece that occurs in the corpus are always kept, so covered-language text never degrades to [UNK]. After truncation, the surviving rows were gently fine-tuned by self-distillation: cosine loss at 1024/256/64 dims against the full model's sentence embeddings, on WebFAQ-v2 and Wikipedia paragraphs of the covered languages (lr 0.03, 3000 steps, batch 1024, sparse Adam on touched rows only). This is a nudge, not a retrain — the median row moves ~10 % of its norm — and it recovers roughly 1–2 nDCG points at small vocabularies; larger learning rates reduce the distillation loss further but hurt retrieval.

Because the vocabulary is smaller, covered-language text tokenizes into ≈21 % more tokens than with the source tokenizer (measured on Wikipedia paragraphs); embedding cost scales with that.

Evaluation

Measured with a local cosine-retrieval harness on the HAKARI-Bench Nano datasets, which reproduces the source card's NanoBEIR-en 53.71 and NanoMIRACL 68.03 exactly. NanoBEIR-en = 13 tasks; MNanoBEIR = 14 languages × 13 tasks (182); NanoMIRACL = 18 languages. All numbers are nDCG@10 × 100, equal weight per task.

Matryoshka truncation

Matryoshka sizes

Caveats

• Losses concentrate in lexically specific tasks (entity names, scientific/biomedical terms) and, at small vocabularies, in languages with long agglutinative words (Telugu, Thai, Bengali), where words fragment into characters.
• Uncased, like the source. Only model.safetensors (fp16) is provided — no ONNX/OpenVINO/int8 exports.

License and credit

MIT, same as the source. All credit for the model and its training goes to the source author (hotchpotch); this repository only removes rows from its table.