papr-embed-code-1-4b

An open-weights text embedding model tuned for code retrieval. It is Qwen/Qwen3-Embedding-4B further trained on code-retrieval data; it takes text and returns one dense vector. It does not take a schema (for schema-conditioned embeddings see papr-embed-schema-1).

What it is good at, and what it is not

• Good at: code retrieval (natural-language to code, code to code, SQL). See CoIR below.
• Not a general-purpose upgrade. On non-code retrieval it is flat to worse than the Qwen/Qwen3-Embedding-4B it starts from, with a clear drop on TREC-COVID (table below). If your workload is mostly prose, use the base model.

Usage

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("papr-ai/papr-embed-code-1-4b")  # loads in fp32
q = model.encode(["Instruct: Given a question, retrieve the code that answers it\nQuery:how do I reverse a list in python"], normalize_embeddings=True)
d = model.encode(["def rev(xs): return xs[::-1]"], normalize_embeddings=True)
print(q @ d.T)

Queries take an instruction in the form Instruct: {instruction}\nQuery:{query}; documents are embedded bare. The instructions behind every number below are in task_prompts.json.

Hosted: POST https://memory.papr.ai/v1/embeddings with "model": "papr-embed-code-1-4b".

Evaluation

Run with stock mteb 2.10.0, fp32, max length 8192, on 8x H100 80GB. The full protocol, including every deviation and when it was decided, is in PROTOCOL.md in this repo.

CoIR (nDCG@10). Mean over the 10 tasks: 80.13

Batch size is mteb's default (32) wherever the task fit in 80 GB in fp32; tasks that ran out of memory were rerun at 16. Scores on most tasks barely move with batch size. SyntheticText2SQL does: the same weights scored between 65 and 72 depending on batching and hardware, because the corpus contains near-identical SQL answers and which one ranks first comes down to floating-point ties. Treat differences of a few points on that task as noise. For reference, stock Qwen3-Embedding-4B scores 68.44 on it in this same environment (its leaderboard entry, from an older mteb, is 78.21).

Held out from training (nDCG@10, fp32).

SciFact: 86.79 vs 78.33 for the base. We trained on the SciFact train split, so this is an in-domain result, not zero-shot.

Training data and contamination

No test split was used anywhere; only train splits entered training. The model saw the train splits of 8 of the 10 CoIR tasks (CodeFeedback-ST and CodeTransOcean-Contest were never trained on), plus the SciFact train split, and its MTEB metadata declares these as training data. CoIR is therefore mostly an in-domain result for this model, not a zero-shot one. In the held-out table, MTEB's CodeSearchNet shares its source with CoIR's CodeSearchNet, whose train split we did use, so read that row as near-domain. CodeEditSearch was not confirmed absent from our training mix either, so read it as indicative.

Limitations

• Regresses on non-code retrieval (above). - Running in fp16 costs about 0.8 points. - English and code only; not evaluated on other languages.

Reproduce

replicate_mteb.py in this repo is the script behind every number above. With mteb==2.10.0, torch==2.10.0, transformers==4.57.6, sentence-transformers==5.2.3:

hf download papr-ai/papr-embed-code-1-4b --local-dir papr-embed-code-1-4b
python papr-embed-code-1-4b/replicate_mteb.py --model papr-embed-code-1-4b --tasks coir --out results --batch 32 --efficient-attn

--efficient-attn only changes which attention kernel runs (same math, ~1e-7 relative). On an 80 GB GPU in fp32, Apps, StackOverflowQA, CodeFeedback and both CodeSearchNet tasks need --batch 16.