NightOwl-CodeEmbedding-v2 🦉

NightOwl-CodeEmbedding-v2 is a compact, 768-dimensional dense embedding model built for code retrieval, code-edit retrieval, and technical question answering.

It is an updated version of Shuu12121/NightOwl-CodeEmbedding (v1). Compared with v1, it:

• supports inputs of up to 4,096 tokens (v1: 1,024), and
• is much stronger on competitive-programming retrieval (AppsRetrieval 0.392 → 0.518) and long multi-turn queries (CodeFeedbackMT 0.767 → 0.886),

while staying within about 0.01 of v1 on every other task (see MTEB Results).

The model uses CLS pooling with cosine similarity and does not require query: / passage: style prefixes.

What changed from v1

1. Longer context (1,024 → 4,096 tokens). v1 was trained with a 1,024-token limit, but simply raising the limit to 4,096 — with no further training — improves long-input tasks without hurting any task (MTEB 2.5.1, 11 tasks):

   Task	v1 @ 1,024	v1 @ 4,096
   CodeFeedbackMT	0.7669	0.8870
   CodeTransOceanContest	0.7595	0.7919
   CodeTransOceanDL	0.3606	0.3747
   AppsRetrieval	0.3918	0.3999
   Other 7 tasks		+0.000 to +0.004
   Mean of 11 tasks	0.6987	0.7152

   About 66% of CodeFeedbackMT queries exceed 1,024 tokens, and the question to answer is usually in the last turn, which v1 truncated away.

2. Additional fine-tuning + weight interpolation (WiSE-FT). v1 was fine-tuned further on competitive-programming data plus a replay of its own training data (see Training). The released weights are the average of v1 and the fine-tuned weights (α = 0.5). The fully fine-tuned model reached a higher AppsRetrieval score (0.540) but lost 0.02–0.06 on several other tasks. Interpolating at α = 0.5 keeps most of the gain while removing almost all of the regressions.

Highlights

• Compact (150.8M parameters), 768-d single-vector embeddings: plain dot-product / cosine search that drops straight into any vector-database stack
• 4,096-token context
• 0.7356 mean NDCG@10 on the 12 MTEB(Code, v1) tasks (self-evaluated with MTEB 2.5.1; v1's leaderboard score is 0.7091)
• Covers eight programming languages: the six CodeSearchNet languages plus Rust and TypeScript
• Handles NL-to-code search, code-to-code retrieval, code-edit retrieval, and technical QA
• Decontaminated against the benchmark test sets used for evaluation (see Data Decontamination)
• Drop-in compatible with sentence-transformers; Apache-2.0 licensed

Supported Languages

• Go, Java, JavaScript, PHP, Python, Ruby (CodeSearchNet languages)
• Rust, TypeScript (additional)

Performance on languages outside this set is untested and may vary.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Shuu12121/NightOwl-CodeEmbedding-v2")
print(model.max_seq_length)  # 4096

queries = ["Python function that sorts a list in descending order"]
documents = [
    "def sort_desc(values): return sorted(values, reverse=True)",
    "def average(values): return sum(values) / len(values)",
]

query_embeddings = model.encode(queries)
document_embeddings = model.encode(documents)

# Cosine similarity (embeddings are normalized internally by similarity())
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)

If memory or latency matters more than long-input quality, you can lower the limit, e.g. model.max_seq_length = 1024.

Model Details

MTEB Results

Evaluation setup:

• MTEB version: 2.5.1
• Metric: NDCG@10
• Hardware: NVIDIA RTX PRO 6000 Blackwell
• Batch size: 64
• max_seq_length: 4,096 (v2), 1,024 (v1, its released setting)

Multi-subset task scores are reported as macro averages. v1 was re-evaluated with the same MTEB version, so the two columns are directly comparable.

¹ CodeEditSearchRetrieval does not provide a standard test split in MTEB, so the official train split is used for evaluation. These examples were not used for fine-tuning. See Data Decontamination.

² v1 was not re-evaluated on COIRCodeSearchNetRetrieval (about 1M documents). This value is from v1's model card (MTEB 2.15.1), so it is not strictly comparable.

The CoIR macro average uses the same 10 tasks as v1's card (all 12 except CodeEditSearchRetrieval and CodeSearchNetRetrieval).

Leaderboard

v1 is listed on the MTEB(Code, v1) leaderboard with a task mean of 70.91. v2's results above are self-evaluated and have not yet been submitted to the leaderboard.

As with v1, MTEB(Code, v1) is largely in-domain for this model (8% zero-shot): it was trained on most of these task families, so its scores reflect in-domain retrieval rather than zero-shot transfer. Comparisons with models that are 100% zero-shot (e.g. embeddinggemma-300m, Qwen3-Embedding) are not apples-to-apples.

Comparison with similar-sized models

Leaderboard task means (×100) for compact dense single-vector models. v2's value is self-evaluated (MTEB 2.5.1) and is shown for reference.

† Self-evaluated with MTEB 2.5.1; not an official leaderboard entry.

Training

v1

v1 was fine-tuned from Shuu12121/NightOwl with CachedMultipleNegativesRankingLoss (bidirectional query-to-document and document-to-query) on 2,534,400 samples, each with one positive and 15 hard negatives mined by Qwen/Qwen3-Embedding-0.6B (1 epoch, learning rate 6e-5, batch size 1024, max sequence length 1,024). The data covers the CoIR task families of MTEB(Code, v1), custom code–comment pairs in eight languages, and code-edit data derived from commitpackft. See the v1 model card for details.

v2

Starting from v1, the model was fine-tuned on:

1. Competitive programming (problem statement → Python solution): APPS train and the CodeContests train split (up to 3 Python solutions per problem).
2. Code translation (Python → C++): one (Python, C++) solution pair per CodeContests problem, targeting CodeTransOceanContest.
3. Replay of v1's training data to limit forgetting: samples from coir_hard_negative_datasets_v3_kd (CodeFeedbackMT 4,000, CodeFeedbackST 3,000, StackOverflowQA 3,000, CosQA 1,000, SyntheticText2SQL 2,000, CodeTransOceanContest 561) and codeedit_hard_negative_datasets_kd (300 per language, 13 languages).

For data 1 and 2, 7 hard negatives per example were mined with Qwen/Qwen3-Embedding-0.6B; solutions of the same problem (or of near-identical problems) were excluded from the negatives. Data 3 reuses the hard negatives already in the datasets.

Interpolation results (MTEB 2.5.1, 11 tasks, max_seq_length 1,024):

Data Decontamination

Overlaps with the evaluation data were removed before training.

v1:

• Custom code–comment pair data vs. the CodeSearchNet test split
• commitpackft-derived code-edit data vs. the CodeEditSearchRetrieval evaluation data

v2 (additional data):

• APPS train problems whose statement has cosine similarity ≥ 0.95 to an APPS test problem
• CodeContests problems that are the same Codeforces problem as an APPS test problem, and problems whose statement has cosine similarity ≥ 0.90 to any APPS test or APPS train problem
• Python → C++ pairs whose Python or C++ code has cosine similarity ≥ 0.95 to a CodeTransOceanContest test query or its answer

Similarities were computed with v1. The replay data is a subset of v1's training data, so it adds no new overlap.

For CodeEditSearchRetrieval, note that MTEB labels the evaluation split train. This is just the official split name for the task; the evaluated examples were not included in this model's fine-tuning data. The score should therefore be read as in-domain generalization on held-out benchmark examples.

Base Model: the NightOwl Backbone

The backbone, Shuu12121/NightOwl, is a ModernBERT-style code encoder pre-trained from scratch, tokenizer included.

• Code-aware tokenizer: a custom 50,368-token BPE tokenizer in which whitespace is tokenized independently of adjacent words, so indentation gets its own tokens and the vocabulary is not wasted on "indent + word" variants.
• Two-phase pre-training with masked-language modeling (mlm_probability = 0.3): standard random-token MLM over code, natural language, and documentation (Phase 1), then line-level MLM on code only, masking whole source lines (Phase 2).

See the v1 model card for the pre-training data and the backbone comparison.

Intended Use

• Natural language to code search
• Code-to-code retrieval and similar-function search
• Code-edit retrieval (matching edit intents to code changes)
• Retrieval over programming Q&A, long multi-turn technical conversations, and competitive-programming problems
• Local semantic code search systems
• RAG systems over codebases and developer documentation

Limitations

• Specialized for code-related retrieval; may underperform general-purpose text embedding models on unrelated natural-language tasks.
• Inputs longer than 4,096 tokens are truncated. The model was trained on inputs of at most 1,024 tokens and the backbone was pre-trained on at most 2,048; the 4,096 setting is supported by the evaluation results above, but quality on very long inputs is less tested than on short ones.
• Slightly lower than v1 on CodeSearchNetCCRetrieval (−0.009) and CodeSearchNetRetrieval (−0.002).
• MTEB(Code, v1) is largely in-domain for this model (8% zero-shot). On code domains, query styles, or languages far from the training distribution, expect lower performance than the benchmark numbers suggest.
• The model produces dense single-vector embeddings. For very fine-grained token-level matching, late-interaction models or cross-encoder rerankers may be worth considering.

Recommended Indexing Settings

Encode both queries and documents with normalized embeddings:

embeddings = model.encode(texts, normalize_embeddings=True)

With normalized embeddings, dot product is equivalent to cosine similarity.

For codebase search, indexing function-level or class-level chunks usually works best.

Citation

If you use this model, please cite it together with the base model and Sentence Transformers.

@misc{nightowl_codeembedding_v2,
  title = {NightOwl-CodeEmbedding-v2},
  author = {Shuu12121},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Shuu12121/NightOwl-CodeEmbedding-v2}
}