m2v-qwen3-4b-256-edu-full
A Model2Vec static embedding model for German education content,
distilled from Qwen/Qwen3-Embedding-4B and fine-tuned on German education texts.
Training data: the synthetic dataset JanSchachtschabel/synthetic-german-educational-materials-50k (CC BY 4.0; LLM-written German education texts); a keyword list (single terms, no texts from its source).
Usage
from model2vec import StaticModel
model = StaticModel.from_pretrained("JanSchachtschabel/m2v-qwen3-4b-256-edu")
embeddings = model.encode(["Bruchrechnung für die 6. Klasse", "Photosynthese"])
The added vocabulary holds inflected forms, base forms, lowercase variants of corpus words and keywords. The model can be used on raw text and on noun/base-form input (e.g. "Brüche kürzen" -> "Bruch").
Training data
How each part of the data was used:
| Data | Vocabulary | Tokenlearn (teacher embeddings as targets) | Pair fine-tuning |
|---|---|---|---|
| Titles and descriptions | words | one text per document: "title. description. keywords" | title -> description |
| Keywords of the documents | keywords used by at least 3 documents | inside the document text | keyword -> its document (title and description, without keywords) |
| Full texts | words | Markdown removed, split into chunks of up to 1,200 characters | searched for keyword-list terms |
| Metadata labels | words | one text per document from subject, topic, target group, level, content type and educational context | – |
| Questions (all per document) | – | one text per question | question -> its document (title and description) |
| Answers | – | one text per answer | question -> its answer |
| Keyword list | every term | every term, as written, lowercase, as noun base forms | term -> up to 3 documents that mention it |
- Split: 5% of the documents were held out for the evaluation; their texts were used in no training step. Records of one topic and near-duplicate titles or descriptions are always on the same side.
- Vocabulary: words that occur at least 5 times, plus their base forms and lowercase variants; at most 250,000 entries are added to the teacher's tokenizer.
- Tokenlearn: 500,000 document texts (randomly sampled from the texts above) plus the keyword-list terms. The targets are the teacher's embeddings of the original texts; 50% of the inputs were reduced to noun base forms, so the model works on both spellings.
- Pair fine-tuning: 500,000 pairs, randomly sampled from all pairs above, contrastive with in-batch negatives.
- Not used: education sector, text type, media form, didactic function, working time and the question types.
Training
- Teacher:
Qwen/Qwen3-Embedding-4B - Dimensions: 256, quantization: float16
- Training input mode: mixed (share of noun/base-form texts: 0.5)
- Tokenlearn: yes, pair fine-tuning: yes
- synthetic dataset records: 50,000
- keyword-list terms: 24,667
- training documents: 47,452
- held-out test documents: 2,548
- added vocabulary entries: 249,072
- tokens in the final vocabulary: 479,325
- tokenlearn examples: 553,109
- training pairs: 500,000
Evaluation
Held-out test split, documents kept apart from training by title and description. "raw" feeds texts unchanged, "lemma" reduces queries and documents to noun base forms first.
| Model | Input | keyword_ndcg@10 | keyword_recall@10 | keyword_lower_ndcg@10 | title_mrr@10 | title_recall@10 | question_mrr@10 | subject_macro_f1 |
|---|---|---|---|---|---|---|---|---|
| this model | raw | 0.494 | 0.576 | 0.379 | 0.631 | 0.885 | 0.416 | 0.715 |
| this model | lemma | 0.480 | 0.563 | 0.476 | 0.588 | 0.851 | 0.391 | 0.736 |
| tfidf | raw | 0.463 | 0.469 | 0.463 | 0.646 | 0.825 | 0.385 | 0.794 |
| tfidf | lemma | 0.517 | 0.533 | 0.517 | 0.619 | 0.820 | 0.459 | 0.810 |
| JanSchachtschabel/m2v-gte-256-edu-v2 | raw | 0.533 | 0.616 | 0.415 | 0.701 | 0.950 | 0.432 | 0.726 |
| JanSchachtschabel/m2v-gte-256-edu-v2 | lemma | 0.523 | 0.599 | 0.519 | 0.670 | 0.920 | 0.452 | 0.708 |
| minishlab/potion-multilingual-128M | raw | 0.385 | 0.463 | 0.334 | 0.602 | 0.876 | 0.329 | 0.681 |
| minishlab/potion-multilingual-128M | lemma | 0.394 | 0.479 | 0.393 | 0.569 | 0.857 | 0.349 | 0.656 |
- keyword_*: each keyword is a query, relevant are the documents tagged with it (documents without keywords)
- title_*: find a document's description from its title
- question_*: find a document from a question about it (where the data has questions)
- subject_macro_f1: logistic-regression probe on the embeddings, single-subject documents
Training data and attribution
This model was trained on Synthetic German Educational Materials (50k) by JanSchachtschabel, licensed under CC BY 4.0. Used fields: titles, descriptions, keywords, full texts, questions, answers, metadata labels (subject, topic, target group, level, content type, educational context).
Changes: the texts were cleaned of Markdown, split into chunks, partly reduced to noun base forms and embedded with the teacher model; words and keywords of the texts were added to the vocabulary. The model holds token vectors learned from this data, not the texts themselves.
The dataset's metadata and topics come from these sources, whose attribution is passed on here:
- OpenEduHub metadata vocabularies (CC0 1.0), including the higher-education subject classification (Quelle: Statistisches Bundesamt (Destatis)) and KldB 2010 (Quelle: Statistik der Bundesagentur für Arbeit)
- Gemeinsame Normdatei (GND) via lobid-gnd, hbz union catalogue via lobid-resources (CC0 1.0)
- OpenAlex topics (CC0 1.0)
- ESCO classification (Quelle: ESCO, Europäische Kommission; reuse under Commission Decision 2011/833/EU)
- Volkshochschul-Statistik, DIE: Echarti, Huntemann, Lux, Reichart (2022), DOI 10.3278/70681 (CC BY 4.0)