litil-embed-0.6b

A 0.6B legal retrieval embedding model. It is Octen/Octen-Embedding-0.6B fine-tuned on legal retrieval data and distilled from Octen/Octen-Embedding-8B, then merged back toward the base. Same architecture, tokenizer, prompts and 1024-dim output as the base, so it is a drop-in replacement.

MTEB(Law, v1)

nDCG@10, mteb 2.20.5, same prompts and settings as the base.

Task	Octen-Embedding-0.6B	litil-embed-0.6b
AILACasedocs	59.95	56.28
AILAStatutes	89.60	89.53
GerDaLIRSmall	41.08	45.20
LeCaRDv2	73.82	73.96
LegalBenchConsumerContractsQA	81.41	82.03
LegalBenchCorporateLobbying	94.77	95.64
LegalQuAD	65.82	70.45
LegalSummarization	72.86	73.34
Mean	72.41	73.30

Gains are largest on German legal retrieval. AILACasedocs is below the base; the training set has no Indian case law.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("litillabs/litil-embed-0.6b")

queries = ["Instruct: Retrieve text based on user query.\nQuery: Kündigungsfrist bei befristetem Mietvertrag"]
documents = ["§ 542 BGB Ende des Mietverhältnisses ..."]

q = model.encode(queries, normalize_embeddings=True)
d = model.encode(documents, normalize_embeddings=True)
scores = q @ d.T

Queries take an instruction prefix (Instruct: ...\nQuery: ); documents take none. Use the same instructions as the base model for a given task. Cosine similarity. Context up to 32k tokens; we evaluated at 18,480.

Training

Base: Octen/Octen-Embedding-0.6B (revision 1a00a4e8).
Teacher: Octen/Octen-Embedding-8B (revision 5adcfa29), used to mine hard negatives and to provide relevance scores for distillation.
Objective: InfoNCE with 4 teacher-mined hard negatives per query plus in-batch negatives, and a KL term against the teacher's score distribution.
LoRA rank 32 on attention and MLP projections, batch 64, learning rate 2e-5, 855 steps, 35% general-retrieval replay. The adapter was merged into the base at half strength.

Data

Source	Rows	Licence
GerDaLIR train split	20,000	MIT
CUAD-QA, PolicyQA, PrivacyQA (question → clause)	11,993	CC-BY-4.0, MIT, MIT
BillSum (title → summary)	8,000	CC0
LeCaRDv2 train split, LEAD	2,554	MIT
German federal statutes (heading → section)	2,150	ODbL
Natural Questions, GermanQuAD, MIRACL zh (general replay)	10,000	CC-BY-SA-3.0, CC-BY-4.0, Apache-2.0

All rows were checked against the eight MTEB(Law, v1) test sets for exact and near-duplicate overlap before training. GerDaLIR and LeCaRDv2 train splits share source corpora with GerDaLIRSmall and LeCaRDv2; the model is therefore not zero-shot on those two tasks and declares both as training datasets.

Limitations

Trained for English, German and Chinese legal retrieval. Indian case retrieval is weaker than the base. Not trained for classification, clustering or STS.

Licence

Apache-2.0, same as the base model.