mT5-small – Metadata for German Educational Materials (Test Model)

Test and research model — not for production use. The model generates titles, descriptions and keywords for German educational materials. Its quality is sufficient for a prototype, but not for unchecked use in production, editorial systems or automated decisions. Titles are still the weakest field (see Evaluation and Weaknesses), although version 0.2-test improved them substantially over the previous release. All outputs must be reviewed by a human.

Demo

Try the model in the browser: mT5 Metadata Demo.

The Space runs on ZeroGPU and offers five sample texts from the test split plus free text input. You can generate the title, description and keywords individually or all three at once, and compare the result with the reference metadata. It is a test UI, not a production tool: outputs still require human review. Free ZeroGPU usage is subject to a daily quota (2 min/day unauthenticated vs. 5 min/day with a free Hugging Face account), so log in for the larger quota.

Overview

This model is a Seq2Seq model based on mT5-small that generates three metadata fields from the full text of a German-language educational resource:

• title
• description
• keywords

All three targets are learned by one shared model. A prefix in the input text (title:, description:, keywords:) selects which field is generated. The model was fine-tuned on synthetic educational materials and is specialised in German educational language.

Version history

Training data

The dataset contains 50,000 synthetic German educational materials from all education sectors (school, higher education, vocational education and training, adult education, early childhood education). Each record consists of a full text of about one A4 page and the fields title, description and keywords derived from it.

The splits are separated by topic: all materials generated from the same topic are in the same split, so the model sees no training topics during evaluation. The texts were generated by a language model; they are not real, expert-reviewed materials (see the dataset's limitations).

The current dataset revision is 38817be749110314ab7be6508c5b4055e291bc73 (2026-11-05). It replaced the titles: 99.8 % of them differ from the previous dataset version. The new titles are shorter, natural editorial titles (median 45 characters, 5 words; 99.4 % have 3–8 words; 28.4 % contain article forms). The model was trained from scratch on this revision; the previous checkpoint was not resumed. About 0.2 % of the titles are unchanged from the previous version.

The three training targets

Each prefix is followed by a single space. The prefix is placed directly before the full text, e.g. title: <full text>. The input is truncated to 1024 tokens (mode head, i.e. the beginning of the text). All settings are stored in the repository in metadata_generation.json.

Usage

import json

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "JanSchachtschabel/mt5-small-german-educational-metadata"

tok = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)
device = "cuda" if torch.cuda.is_available() else "cpu"
if device == "cuda" and torch.cuda.is_bf16_supported():
    model = model.to(torch.bfloat16)  # bf16 as in training; fp16 is unstable with mT5
model = model.to(device).eval()

settings_path = hf_hub_download(repo_id, "metadata_generation.json")
with open(settings_path, encoding="utf-8") as fh:
    settings = json.load(fh)
text = """The full text of the educational material goes here ..."""

for task, prefix in settings["tasks"].items():
    enc = tok(prefix + text, max_length=settings["max_source_length"],
              truncation=True, return_tensors="pt").to(device)
    with torch.no_grad():
        ids = model.generate(
            **enc,
            max_new_tokens=settings["max_target_lengths"][task],
            num_beams=4,
            no_repeat_ngram_size=3,
        )
    print(f"{task}: {tok.decode(ids[0], skip_special_tokens=True)}")

Notes:

• Input: use clean plain text where possible (no navigation, cookie notices or HTML residue). During training, Markdown was reduced to plain text and the text was cut after 1024 tokens.
• Beams: num_beams=4 is the default. 2 is usually similarly good and slightly faster; 1 (greedy) is the fastest but tends to repeat phrases and invent words. 8 brings little improvement for this model.
• Length: for long texts only the beginning is used. For long web pages, trim to the relevant sections first.
• Review: outputs are suggestions, not verified metadata.

Training

The final model corresponds to the best checkpoint (checkpoint-135432, epoch 4, validation loss 2.645).

The three targets contribute unequally to the token-level loss: descriptions dominate, keywords come second, and titles contribute the smallest share. The model therefore optimises mostly for descriptions; titles benefit least from the token-level objective.

Evaluation

Evaluated on the first 200 materials of the test split (2,393 materials, topics not seen in training), 4 beams, bf16, with the same preprocessing and length limits as in training. Both versions were evaluated on the same 200 materials of the current test split:

Metric definitions:

• Title strict: exact match (case-insensitive).
• Title loose: token F1 between generated and reference title.
• Description: word-level ROUGE-L F1.
• Keywords strict: F1 over the comma-separated keyword lists (whole terms).
• Keywords loose: token F1 over all keyword words.

Interpretation: the new titles raise exact match from 0.5 % to 8.0 % (16×) and word-overlap F1 from 0.419 to 0.459. Descriptions are essentially unchanged (0.296 → 0.299); keywords improve in both strict and loose scores. Titles remain the weakest field: 8 % exact match is still low, and the model usually paraphrases the reference instead of reproducing it. Exact match is a strict metric for natural editorial titles, but it is the metric the dataset supports. Remaining errors include repetitions and occasional grammar slips.

Strengths

• Titles improved substantially over the previous version: 8.0 % exact match and 0.459 word-overlap F1 on the test split.
• Produces fluent, topically appropriate descriptions for German educational texts.
• Usually returns plausible keywords from educational and subject-specific vocabulary.
• A small model (300M parameters) that runs on a single GPU or CPU.
• All three fields with one model and a uniform prefix scheme.

Weaknesses and limitations

• Titles are still the weakest field. Exact match is 0.080 and word-overlap F1 is 0.459; most titles are topically appropriate but phrased differently. Titles should not be used without review. Beam search (2–4) improves quality over greedy decoding but does not solve the problem.
• Repetitions and grammar slips occur in descriptions (e.g. duplicated words such as „Datenqualität, Datenqualität“ and occasional subject-verb agreement errors).
• Synthetic training data: the texts come from a single language model and have a uniform style. The model may invent facts or reuse formulations that are not supported by the specific material.
• Domain shift: real websites, scanned scripts or very long documents differ from the training data. Inputs are truncated after 1024 tokens; the rest is ignored.
• German only / German education system. Other languages or education systems are not supported.
• No human evaluation: all metrics come from the synthetic test split. Quality on real materials has not been systematically evaluated.
• Not production-ready: no guarantee of correctness, completeness, consistency or freedom from copyright issues in the outputs. Do not use for automated grading, selection or other high-stakes decisions.

License and provenance

• Model weights: MIT, derived from the MIT-licensed base model deutsche-telekom/mt5-small-sum-de-mit-v1.
• Training data: JanSchachtschabel/synthetic-german-educational-materials-50k, licensed under CC BY 4.0. The data is synthetic; the dataset card describes its origin, limitations and license terms in detail.
• Title targets: regenerated for dataset revision 38817be with the project's updated title prompt; 99.8 % of the titles differ from the previous dataset version.
• Base model provenance: the base model is a German summarization model by Deutsche Telekom AG, based on google/mt5-small.

Feedback

Errors, unsuitable outputs or suggestions can be reported through the Discussions of this repository. Production use would require a larger, curated dataset, a better title metric (e.g. keyword coverage instead of exact match) and a human evaluation.