SKILLRET-Edge-109M-int3

Engineering blog: Skill Retrieval on a CPU: Building the SKILLRET Small Model Family (한국어: CPU에서 도는 스킬 검색)

A 109.5M-parameter bi-encoder for agent skill retrieval — picking the right skill out of a catalogue for a natural-language request. Distilled from ThakiCloud/SKILLRET-Embedding-0.6B and small enough to run on a CPU next to the agent.

Scored on the public ThakiCloud/SKILLRET test split (4,392 queries / 6,006 skills), the metric the SkillRet paper uses as its headline.

NDCG@10 78.04 · 68.4 MB packed

Correction — September 2026. Earlier versions of this card reported the SKILLRET-Embedding-0.6B teacher as 80.82 NDCG@10 on the current 4,392-query / 6,006-skill evaluation split. That score was measured before the train/evaluation query-prefix contract was corrected. Re-evaluation on ThakiCloud/SKILLRET revision a050ad2 under the canonical query contract gives 78.48 NDCG@10. The model weights are unchanged. The earlier 605 MB size was also not the serialized artifact size; the distributed BF16 checkpoint is 1191.6 MB (decimal MB). The "vs teacher" column was also still computed against 80.82 in earlier versions of this card; it is now computed against 78.48, the teacher score measured under the same evaluation condition as the rows below. We retain this note so results quoted from earlier versions of the card can be interpreted correctly. Correction — November 2026: two rows of "What did NOT work" (8B-teacher distillation, hard-negative mining) are now marked as past experiments not re-verified under the current evaluation contract; statements of a statistical tie or advantage that were not backed by a paired test were removed; and where a teacher latency appeared, it is now reported separately from the student latency with its own measurement conditions.

Results

Variant	Size on disk	NDCG@10	vs teacher
SKILLRET-Embedding-0.6B (teacher)	1191.6 MB	78.48	—
fp16	219.0 MB	79.18 ± 0.42	100.9%
int8 / g16	136.9 MB	79.18 ± 0.42	100.9%
int4 / g16	82.0 MB	79.21 ± 0.42	100.9%
int3 / g16	68.4 MB	78.04 ± 0.44	99.4%

On SkillRet the 109M model's observed score of 79.18 is 0.70 points above the 0.6B teacher's 78.48, but whether that is a statistically significant advantage has not been established. Both scores were measured on the same split (ThakiCloud/SKILLRET revision a050ad2, 4,392 queries / 6,006 skills, NDCG@10), each with the query format the model was trained on: the teacher with its instruction prompt, this model with no prefix. The ±0.42 in the table is the standard error of this model's own mean over queries, not of the difference between the two models, and no paired test against the teacher was run. The "vs teacher" column is a plain ratio, not a significance claim. int8 and int4 are free; int3 costs 1.14pp and is the point where quantization starts to be a real trade.

Usage

from sentence_transformers import SentenceTransformer
m = SentenceTransformer("ThakiCloud/SKILLRET-Edge-109M-int3")

# ⛔ Encode queries BARE — no instruction prefix. See query_prefix.json.
q = m.encode(["I need to convert a spreadsheet into a chart"], normalize_embeddings=True)
d = m.encode(["Chart Builder — turn tabular data into bar/line charts"], normalize_embeddings=True)
print(q @ d.T)

What "int3 / g16" means here

Weights are quantized group-wise, asymmetric min/max, group size 16, with scales and zero-points stored in fp16 — the same shape as a GGUF Q4_K/Q8_0 block.

This repo ships two things, deliberately:

File	What it is
`model.safetensors`	The quantized values, de-quantized back into fp16 so any `transformers` / `sentence-transformers` install loads it today. This is the file that produces the score above.
`model-int3-g16.bin`	The actual packed payload — 68.350 MB. This is what the size claim means.

Be clear about what this does and does not buy you. The packed file is the real storage footprint and it is verified: predicted 68.594 MB vs 68.350 MB on disk (0.36% error). But the safetensors path runs fp32 arithmetic — the latency figures below were measured that way, and they are not integer-kernel speeds. Routing the packed weights through a real INT kernel (llama.cpp, ONNX Runtime) is unmeasured and is where further speedup would come from.

quantization.json carries the per-tensor layout if you want to write that kernel.

How it was trained

Knowledge distillation from ThakiCloud/SKILLRET-Embedding-0.6B (NDCG@10 78.48 on the same split), kd_weight=0.7, multi-positive InfoNCE over 1–3 golds per query, 12 epochs, cosine schedule. The epoch was chosen on a skill-disjoint holdout — the test split was never used for selection.

⚠️ Read this before you compare numbers

Use the same query prefix at train and eval time. This repo ships query_prefix.json recording the contract (resolved: "", i.e. queries are encoded bare, with no instruction prefix). Which prefix you pick made little difference after fine-tuning in our runs — none 79.18 vs an instruction prefix 78.50 for the 109M student, a 0.68-point gap we did not test for significance — but keeping it consistent matters a great deal. We measured a +8.84pp swing on a single checkpoint purely from a train/eval prefix mismatch, and that mismatch also manufactured a fake result in which quantization appeared to beat fp16. It does not.

Other caveats worth stating plainly:

The model card of the SkillRet reference models reports a 4,997-query / 6,660-skill split. That split is not in the currently published dataset — we verified the public files are hash-identical to ours at 4,392 / 6,006. Do not convert between the two.
Standard error on this split is about ±0.45. Differences under ~1pp are not rankings.
Pooling is CLS, embeddings are L2-normalized, max_length=256 at evaluation.

What did NOT work (so you don't repeat it)

Attempt	Result
Distilling the 22M student from the 8B teacher instead of 0.6B	72.86 vs 75.26 NDCG@10 (−2.40pp observed). Past experiment, not re-verified under the current evaluation contract: it comes from a ledger that has since been superseded, and the two runs used different batch sizes (64 vs 128), so it is not a clean control. Not measured for the 109M student. Not a statement about current performance
INT2 / ternary post-training quantization of the 0.6B teacher (SKILLRET-Embedding-0.6B, not these students)	Total collapse (~0.1 NDCG@10, the level of random retrieval at 0.086) — GPTQ and QuIP do not rescue it. Measured 2026-09-09 under the earlier query-prefix contract
INT2 / g16 post-training quantization of the students	Not a collapse, but a large loss: 22M 59.42 (fp16 75.26), 109M 63.55 (fp16 79.18), same evaluation condition as the Results table
Hard-negative mining (our mined set, 22M student)	74.17–74.26 vs 75.26 NDCG@10 (about −1.0pp observed). Past experiment, not re-verified under the current evaluation contract. A defect in the mining step left only 2.49 of the intended 4 negatives per query, so this shows that this mined set did not help, not that hard negatives do not help. The corrected re-run was not completed
LEAF auxiliary projection loss (w=0.3)	−1.51pp (paired t = −7.78)

License

Apache-2.0, inherited from the base model.

Citation

Benchmark: SkillRet, arXiv:2605.05726

Paper (measurements behind the quantized variants and the student frontier): Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families, arXiv:2609.16391

@article{han2026ptqembedders,
  title   = {Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families},
  author  = {Han, Hyojung},
  journal = {arXiv preprint arXiv:2609.16391},
  year    = {2026}
}