An embedding model for finding Dutch legislation and case law from a question in plain words. It is Qwen/Qwen3-Embedding-0.6B with a LoRA adapter merged in, trained on about 280,000 Dutch legal question-passage pairs. Same size, same 1,024 dimensions and same query instruction as the base model.
It was made for the topic search in LexAgent, which gives AI assistants access to the official Dutch and EU legal sources. The unmerged adapter is SWIS-Leiden/qwen3-embedding-0.6b-nl-legal-lora; both give the same vectors (cosine 0.9996 between them).
The base model reads Dutch well but not legal Dutch. Asked about ontslag op staande voet (summary dismissal), it ranked a tax-collection rule about the beslagvrije voet above article 7:677 of the Civil Code, because both contain the word voet. After training:
| cos(query, BW 7:677) | cos(query, beslagvrije voet) | |
|---|---|---|
| Base model | 0.251 | 0.512 |
| This model | 0.441 | 0.073 |
Article 7:677 is an answer in the held-out exam, so the model never trained on it.
Queries take the base model's instruction (stored as the query prompt); documents take none.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("SWIS-Leiden/qwen3-embedding-0.6b-nl-legal")
queries = ["Mag mijn werkgever mij op staande voet ontslaan?"]
documents = [
"Elk der partijen is bevoegd de arbeidsovereenkomst onverwijld op te zeggen om een dringende "
"reden, onder onverwijlde mededeling van die reden aan de wederpartij.",
"De beslagvrije voet is het deel van het inkomen waarop geen beslag mag worden gelegd.",
]
query_embeddings = model.encode(queries, prompt_name="query")
document_embeddings = model.encode(documents)
print(model.similarity(query_embeddings, document_embeddings))
With vLLM or another OpenAI-compatible server, put the instruction in front of the query yourself:
Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:{query}
The training used exactly this text, so other instructions will work less well.
LexAgent keeps its own search index of Dutch law: 391,000 article passages of regulations in force (wetten.overheid.nl) and 902,000 rulings (Rechtspraak open data), each with the exact text that gets embedded. All training pairs were built from that text, without new requests to the sources.
| Set | Pairs | How |
|---|---|---|
| Questions per article | 141,361 | Qwen3.6-35B-A3B wrote questions for 39,256 article passages, in four styles |
| Questions per ruling | 121,617 | The same model wrote questions for 44,904 rulings, in three styles |
| Judges' citations | 32,395 | A ruling's summary with the citation cut out, paired with the article it cites |
The four styles for an article, here for article 12 of the WAM (car insurance ends when the duty to insure passes to someone else):
| Style | Question |
|---|---|
| Citizen | Ik heb mijn auto verkocht, loopt mijn autoverzekering dan gewoon door of stopt die? |
| Lawyer | Eindigt de motorrijtuigenverzekering van rechtswege bij overgang van de verzekeringsplicht op een ander? |
| Search words | autoverzekering einde verkoop auto |
| Situation | Mijn vader is overleden en ik erf zijn auto; stopt zijn verzekering nu? |
A ruling got the legal question it decides in a lawyer's words, the facts as a citizen would tell them, and a few search words. The prompt forbids naming the law, the article number, the ECLI, the court or the parties, and a question that still names one was dropped afterwards: someone who knows the number does not need this search, and the model would only learn to match numbers. Passages without a real rule (citation titles, commencement articles, repealed articles, pure amendments) were skipped before the model saw them. Half of the laws were used, a quarter of the general administrative orders (AMvB's) and smaller shares of the rest, because people mostly ask about laws. Writing the questions took about 86,000 requests and 85 million tokens (temperature 0.7, thinking off), in one night on SWIS's own GPUs.
The citation pairs need no language model. About one in seven rulings cites an article in its summary ("Beroep op artikel 6:248 lid 2 BW verworpen"). The query is the summary's opening sentences with the citation and its lead-in words cut out, the positive is the cited article. A first version was dominated by a few articles (article 8 ECHR alone had 3,330 pairs), so it is capped at 25 pairs per article. Rulings from the Caribbean part of the Kingdom were left out.
Every question also carries up to three hard negatives: texts that look right but are not. They come from the base model's own mistakes. Each question was searched with the base model (hybrid keyword and vector search, top 30), and the hits ranked near or above the right answer became candidates.
A near miss that is actually also right does damage when the model is taught it is wrong, and Dutch law repeats rules often enough for that to happen (the same rule for each group of office holders, ten rulings on one legal question). So:
A real example, for a summary about statutory commercial interest after a lease was dissolved:
| Candidate | Reranker score | |
|---|---|---|
| BW 6:119a, statutory commercial interest | 0.98 | the cited article |
| BW 7:129d, from the title on loans | 0.87 | kept as a negative |
| Book 6 of the BW BES | 0.52 | kept as a negative |
| A policy rule on statutory interest in payment plans | 0.40 | kept as a negative |
The check was often needed: for 40% of the pairs no hard negative was left, for rulings more than half. Those pairs still train, with random other passages as negatives. They teach less, but nothing false.
That leaves 279,621 training pairs.
The model sees a question with its right passage and has to score that passage above every other passage in the batch: the three hard negatives and the right passages of the other 127 questions. CachedMultipleNegativesRankingLoss computes this over a batch of 128 while holding only 24 in memory at a time, so a large batch fits on one GPU. The batch sampler keeps duplicate questions and passages out of one batch, so a passage is never a negative for its own question.
| Method | LoRA, rank 32, alpha 64, dropout 0.05, on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (about 20M of the 600M parameters), merged afterwards |
| Loss | CachedMultipleNegativesRankingLoss, scale 20, in-batch negatives plus 3 hard negatives per question |
| Batch | 128 (mini-batches of 24), no duplicates within a batch |
| Schedule | 1 epoch over 279,621 pairs, learning rate 1e-4 with 5% warmup, bf16, max 512 tokens, seed 42 |
| Query input | The base model's instruction, exactly as the production embedding service sends it |
| Hardware | one NVIDIA L40S, shared with production models, 14.8 hours |
| Software | sentence-transformers 4.1.0, transformers 4.52.4, peft 0.15.2, PyTorch 2.6 |
LoRA freezes the base model and learns a small correction next to each weight matrix. That keeps what the base model already does well (plain Dutch, plain-language matches) and leaves a 40 MB adapter that can run next to the unchanged base model.
A separate run on the first 50,000 pairs reached most of the gain:
| Training pairs | Recall@10 | MRR@10 |
|---|---|---|
| none (base model) | 0.641 | 0.453 |
| 50,000 | 0.855 | 0.686 |
| 279,621 | 0.900 | 0.748 |
After the first 50,000 pairs it is mostly the order within the top 10 that improves.
Not tested: a run without the hard negatives, and a run without the citation pairs whose cited article the reranker found barely related to the summary (a third of them). Whether either matters is still open.
The dataset itself is not published.
Dev set (5,000 synthetic pairs against their passages plus 20,000 distractors):
| Recall@10 | MRR@10 | nDCG@10 | |
|---|---|---|---|
| Base model | 0.641 | 0.453 | 0.498 |
| This model | 0.900 | 0.748 | 0.785 |
The dev questions come from the same writer as the training questions, which flatters these numbers.
Exam (74 hand-checked questions, searched against the full index of 391,000 legislation passages and 902,000 rulings, hybrid keyword and vector search in Weaviate, each model at its best weighting):
| Legislation (47): top 1 | top 10 | MRR | Rulings (27): top 10 | MRR | |
|---|---|---|---|---|---|
| Base model | 7 | 22 | 0.275 | 7 | 0.161 |
| This model | 20 | 40 | 0.576 | 8 | 0.211 |
The exam is kept private so it cannot leak into training data. Three article questions got worse, among them the general deadline for an objection (article 6:7 Awb), which now ranks below the many laws that set their own deadline in the same words.
Apache 2.0, like the base model and Qwen3.6, which wrote the training questions. Made by SWIS for LexAgent.