dutch-minilm-l6-v2

A small, fast, Dutch-only sentence embedding model distilled from all-MiniLM-L6-v2.

It produces 384-dimensional embeddings in the same vector space as the English teacher model, enabling cross-lingual search and retrieval between Dutch and English.

Model Details

How It Was Made

1. Dutch tokenizer

A new WordPiece tokenizer (30k vocab) was trained from scratch on Dutch text from the parallel training corpora. This gives proper tokenization of Dutch compound words (e.g. "stofzuiger", "aanrechtblad", "vaatwastabletten") instead of fragmenting them into many subword pieces.

2. Student model construction

The student reuses the same MiniLM-L6 transformer architecture (6 layers, 384 hidden size). All transformer layer weights were copied directly from the English teacher. The embedding layer was re-initialized for the new Dutch vocabulary, with shared tokens (punctuation, numbers, overlapping subwords) copied from the teacher.

3. Knowledge distillation

The student was trained to mimic the teacher's sentence embeddings using MSE loss on parallel English-Dutch sentence pairs. The teacher encodes the English sentence, and the student learns to produce the same embedding from the Dutch translation.

Training hyperparameters

• Epochs: 5
• Batch size: 64
• Learning rate: 2e-5
• Warmup ratio: 0.1
• Precision: fp16
• Max sentences per dataset: 500,000

Training datasets (parallel en-nl)

Evaluation

STSb-multi-mt Dutch (unseen, not used during training)

Spearman/Pearson correlation with human similarity judgments on 1,379 Dutch sentence pairs:

STS17 nl-en (cross-lingual, used during training for evaluation)

Spearman/Pearson on 250 Dutch-English sentence pairs:

Per-pair analysis (STSb-multi-mt Dutch test)

• Student better on 56.4% of pairs
• Teacher better on 35.8% of pairs
• About the same on 7.8% of pairs
• Mean absolute error: Student 0.189 vs Teacher 0.251

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MarkSelles/dutch-minilm-l6-v2")

sentences = [
    "De kat zit op de mat.",
    "Een poes ligt op het kleed.",
    "De auto rijdt over de snelweg.",
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 384)

Cross-lingual search (Dutch queries, English corpus)

Since the Dutch student produces embeddings in the same vector space as the English teacher, you can use both models together for cross-lingual retrieval:

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

# Dutch queries
dutch_model = SentenceTransformer("MarkSelles/dutch-minilm-l6-v2")
query_emb = dutch_model.encode(["draadloze koptelefoon"])

# English product catalog
english_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
product_embs = english_model.encode([
    "Wireless Bluetooth headphones with noise cancelling",
    "Wired USB-C earbuds",
    "Portable speaker",
])

similarities = cos_sim(query_emb, product_embs)
print(similarities)

Limitations

• Optimized for Dutch only — will not work well on other languages
• Max sequence length of 128 tokens — longer texts are truncated
• Trained on general-domain parallel data — domain-specific fine-tuning may improve results for specialized use cases
• The student sometimes overestimates similarity for Dutch sentences with overlapping tokens but different meanings

Citation

If you use this model, please cite the original teacher model:

@inproceedings{reimers-2020-multilingual-sentence-bert,
  title = "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation",
  author = "Reimers, Nils and Gurevych, Iryna",
  booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
  year = "2020",
  url = "https://arxiv.org/abs/2004.09813",
}