code-daemon-embed-v1

A 4-layer, 46.8M-parameter code-embedding model built for one job: embedding a whole repository fast enough to re-index it on every commit, as the dense half of a hybrid (vectors + BM25) code search. Short code units — signatures, docstrings, symbol names, one-line descriptions — and short keyword-shaped queries land in one 768-dim space.

What makes it different

• Speed first. 4–6× the throughput of any public 768-dim encoder we tested; a 1.1M-entity C++ repository embeds in about 80 seconds on a laptop GPU.
• A reranker's judgement baked into the vectors. A 4B cross-encoder's ranking was distilled into the bi-encoder at training time; in the system it was built for, a runtime reranker on top of it measured net-negative.
• Nothing to get wrong at the output. Mean pooling and L2 normalization are inside the graph: the model returns unit-norm [B, 768], ready for a dot product.
• INT8 from quantization-aware training, so TensorRT and OpenVINO build an INT8 engine with no calibration pass.

Queries and documents are encoded the same way — no query: / passage: prefix. Hard cap of 128 tokens. English and code only.

Choosing it — the trade

Vector channel alone, every model at the same 128-token budget. Quality is nDCG@10 × 100; speed is texts/s through onnxruntime CUDA FP32, batch 64, at seq 64 (the mean serving text is 70 tokens). Laptop RTX 5060, 2026-09-11.

• 4.0–5.7× the throughput for 1–10 points of nDCG on our queries. Pick it when indexing speed is the constraint and a lexical channel runs beside it; pick gte-modernbert-base when vector-only accuracy is.
• Inside the hybrid search it was built for (80 real agent queries, file level, runtime reranker off): the right file is first 44 % of the time and in the top 5 69 %.
• Not multilingual. The vocabulary prune kept 942 of XLM-R's 31 671 Cyrillic pieces; for non-English queries use multilingual-e5-base.
• Not for long text or general prose. On four CoIR tasks it averages 52.8 nDCG@10 — 89.8 on Python docstring→code, 28.5 on cosqa.

Throughput in a real index

A full index through the Code-Daemon daemon, laptop RTX 5060, TensorRT INT8, pinned 114 W:

The spread between languages is text length: 71 % of C# texts fall in the longest bucket against 24 % for C/C++. Above ~820k texts the daemon adds the Intel iGPU beside the RTX card, so the mysql-server row is a two-device number. The same daemon running multilingual-e5-base compiled to the same INT8 engines is 2.3× slower on every row.

How to use it

Pick a length bucket

Every engine family ships four length buckets. A batch pays for its longest member, so route each text to the smallest bucket that fits it:

• Sort by length, then batch, and dispatch each batch at its own longest length rounded up to a multiple of 8. The TensorRT engines take a dynamic sequence range for exactly this — worth +10 % end to end.
• Batch size is not the lever: 96 × 48 already saturates the GPU, and a bigger batch was slower per text. Running the four engines concurrently does not add throughput either.
• All four TensorRT engines together take ~400 MB of VRAM.
• OpenVINO IRs are static shapes: batch 64 on CPU and iGPU, batch 16 on the NPU. The NPU has the lowest latency per batch (28–95 ms) and suits single interactive queries better than bulk indexing.

Shape the input

Documents — the model was trained on a compact, front-loaded text, and reproducing it is worth more than any inference tuning:

function acquireProjectLock in src/storage/multi_db.zig: zig
path: src storage multi db
[exported]
sig: (allocator: Allocator, project_hash: []const u8) -> !Lock
Acquires the exclusive SQLite lock for one project.

That is {type} {name} in {file}: {lang}, the path as words, flags, the signature, then the first ~200 characters of the doc comment and a one-sentence description; ~768 characters at most. Leave the function body out — it belongs in the lexical channel.

Queries — raw, no prefix. Short keyword bags and behaviour descriptions are what it was tuned for. Longer than 128 tokens — windows of 128, stride 96; mean-pool the window vectors and re-normalize.

Run the ONNX

import onnxruntime as ort, sentencepiece as spm, numpy as np

sp   = spm.SentencePieceProcessor(model_file="sentencepiece.bpe.model")  # pad=0 unk=1 bos=2 eos=3
sess = ort.InferenceSession("model_int8qdt.onnx", providers=["CPUExecutionProvider"])

def embed(texts, max_len=128):
    ids  = [[2, *sp.encode(t)[: max_len - 2], 3] for t in texts]            # bos … eos
    L    = max(len(x) for x in ids)
    inp  = np.array([x + [0] * (L - len(x)) for x in ids], dtype=np.int64)  # pad=0
    mask = (inp != 0).astype(np.int64)
    return sess.run(None, {"input_ids": inp, "attention_mask": mask})[0]    # [B, 768], unit-norm

D = embed(["function acquireLock in src/db.zig: zig\npath: src db"])
Q = embed(["acquire database lock"])
print(Q @ D.T)                                                              # inner product = cosine

Tokenize with SentencePiece — the vocabulary is a unigram model, and a BPE merge loop segments it differently from training. Both inputs are int64.

Building your own engines

Feed model_int8qdt.onnx as it is. TensorRT 11 reads the precision from its Q/DQ nodes; OpenVINO keeps the trained scales as FakeQuantize (run compress_quantize_weights_transformation before saving, or the IR comes out twice the size). Never run a calibration / PTQ pass over it: it replaces the trained scales with fitted ones, and the engine builds and loads cleanly and quietly retrieves worse. Use the bucket profiles from the table above, for example:

trtexec --onnx=model_int8qdt.onnx --saveEngine=embed-m.engine --builderOptimizationLevel=5 \
        --minShapes=input_ids:128x56,attention_mask:128x56 \
        --optShapes=input_ids:128x56,attention_mask:128x56 \
        --maxShapes=input_ids:128x64,attention_mask:128x64

How it was made

• Backbone: intfloat/multilingual-e5-base (XLM-RoBERTa, 12 layers, 278M), cut in depth 12 → 8 → 6 → 4 with a healing pass after each cut.
• Vocabulary pruned to code and English: 250k → 22.7k SentencePiece pieces, which takes the embedding table from 192M to 17.5M parameters — the largest single saving.
• Two dense teachers, mixed: the vector space is a distillation from gte-modernbert-base blended with the text tower of google/embeddinggemma-2 (0.3 / 0.7 of the cosine, projected back to 768-d); Qwen3-Reranker-4B remains the ranking teacher — the student learned the cross-encoder's score distribution over hard negatives, not only "positive above negative".
• Trained on the serving text format itself: 1 546 752 entity texts and queries from 173 repositories, written the way the daemon writes them at index time.
• INT8 by quantization-aware training; the E5 prefixes, the pooler and MLM heads and the token-type input were dropped; pooling and L2 were fused into the export.

Files

A TensorRT engine loads only on the exact GPU architecture and TensorRT version that built it: sm_75 Turing · sm_80 A100/A30 · sm_86 RTX 30xx · sm_89 RTX 40xx/L4 · sm_90 H100/H200 · sm_120 RTX 50xx, all TensorRT 11.3. On sm_120 the m / l / xl engines run attention as one fused fp16 kernel (+11–21 % per bucket); their vectors are interchangeable with the other engines'. If your combination is missing, build it from the ONNX.

License

Released under the MIT license. The backbone (intfloat/multilingual-e5-base) is MIT; the teachers (gte-modernbert-base, Qwen3-Reranker-4B) are Apache-2.0. As is standard practice for distilled embedding models, the weights are released under MIT. Not legal advice.

Attribution

Backbone: intfloat/multilingual-e5-base (MIT). Dense teacher: Alibaba-NLP/gte-modernbert-base (Apache-2.0). Ranking teacher: Qwen/Qwen3-Reranker-4B (Apache-2.0).