code-daemon-denoise-v1

A bilingual (English + Russian) word filter for search vocabularies. Given one word form, it answers whether it is a technical term worth indexing or ballast to drop — the inflected function words, chopped identifier fragments and boilerplate that a harvest of identifiers, comments, docs and commit messages is mostly made of.

What makes it different: frequency cut-offs throw away rare technical terms, which are exactly the ones worth searching for. This model makes the call per word from meaning — and the whole learned decision is 384 numbers and a bias, a single affine on top of a frozen, unchanged multilingual-e5-small. No second model, no scikit-learn at inference: a dot product you can apply in any language. It filters ~800 words/s on a CPU core and ~17 800/s on a laptop GPU.

emb    = session.run(None, {"input_ids": ids, "attention_mask": mask})[0]   # [B, 384], pooled + L2
p_keep = 1 / (1 + np.exp(-(emb @ w + b)))                                   # the whole classifier

Numbers

224 held-out words (95 keep / 129 drop). Like for like, the alternative is the same encoder with a naive head — a logistic fit on seed word lists, which is the supervision you have without a training corpus. Both arms cut at the same number of stripped words (93); AUC needs no threshold at all:

Speeds are onnxruntime CUDA FP32, batch 64 × seq 40.

• The trained head is the whole margin over the same encoder: +0.111 AUC, 8 fewer terms lost.
• The vocabulary prune is free and pays for speed: the pruned and the original encoder score the same AUC (0.7669 vs 0.7674), and the prune is the 1.23× in the words/s column.
• The trade against a bigger encoder: multilingual-e5-base matches it on AUC and loses fewer terms, for a quarter of the throughput and 470 MB against 76 MB. If size and speed do not constrain you, it is a fair alternative.

At the shipped default threshold (0.95) the INT8 engine strips with precision 0.89 and recall 0.77, wrongly dropping 12 of 95 technical words.

Speed by lane

Batch 64 × seq 40, laptop Intel Core Ultra 9 275HX + RTX 5060:

The CPU lane filters a repository's whole harvested vocabulary in seconds without a GPU; CPU, iGPU and NPU together reach ~2 000 words/s.

How to use it

• Single word forms only, prefixed with "vocab: " — the head was trained on prefixed embeddings, so the boundary does not line up without it. Sequence budget 40 tokens.
• Threshold: strip a word only when P(ballast) = 1 − P(keep) ≥ strip_threshold (default 0.95, in denoise_head.json). It is high on purpose — dropping a real term is the expensive error, keeping some ballast is not. Lower it to strip more.
• Feed raw SentencePiece ids. The ONNX bakes in the id offset and the pruned-vocabulary remap.
• Shapes: batch 64 × seq 40 everywhere, except the NPU at batch 16.

import json, numpy as np, onnxruntime as ort, sentencepiece as spm

sp   = spm.SentencePieceProcessor(model_file="sentencepiece.bpe.model")
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
head = json.load(open("denoise_head.json"))            # {dim, w[384], b, strip_threshold}
w, b, thr = np.array(head["w"], np.float32), head["b"], head["strip_threshold"]

def p_keep(words, max_len=40):
    toks = [[2, *sp.encode("vocab: " + x)[: max_len - 2], 3] for x in words]   # bos … eos
    L    = max(len(t) for t in toks)
    ids  = np.array([t + [0] * (L - len(t)) for t in toks], dtype=np.int64)    # pad = 0
    mask = (ids != 0).astype(np.int64)
    emb  = sess.run(None, {"input_ids": ids, "attention_mask": mask})[0]       # [B, 384]
    return 1.0 / (1.0 + np.exp(-(emb @ w + b)))

scores = p_keep(["mutex", "tensorrt", "пожалуйста", "asdfgh"])
strip  = (1.0 - scores) >= thr                          # strip only confident ballast

Not suited to: phrases or sentences; scripts other than Latin and Cyrillic (pruned out); term-vs-stopword calls outside software.

How it was made

• Encoder: intfloat/multilingual-e5-small, frozen — its weights are unchanged. Mean pooling and L2 were fused into the export, and the graph quantized to INT8.
• Vocabulary pruned by script: 250k → 142k pieces (Latin + Cyrillic + punctuation). Nothing outside those scripts was reachable anyway, so it is lossless for the two languages and cuts the INT8 weights from ~121 MB to ~76 MB.
• Head: a logistic regression over the frozen embeddings, folded with its scaler into one (w, b) and temperature-scaled so the threshold stays in charge across its range.

Files

License & attribution

The encoder weights are intfloat/multilingual-e5-small (MIT), redistributed here in compiled form unchanged — MIT permits that redistribution and sublicensing, so this repository ships under Apache-2.0 with the encoder's MIT notice intact. The trained head and the build/quantization tooling are original. Backbone: XLM-RoBERTa. Not legal advice.

Used by the Code-Daemon code assistant, though nothing about the model is specific to it.