cometadata/etd-metadata-parsing-qwen3.5-4b-dapo-v2

A Qwen3.5-4B model for extracting bibliographic metadata from the front matter of electronic theses and dissertations (ETDs). Given the first pages of a thesis (text extracted from the PDF), it emits one 10-field JSON record: title, language, creators, advisors, degree_discipline, degree_grantor, coverage_spatial, degree_department, degree_level, date_issued. Built for the COMET (Collaborative Metadata Enrichment Taskforce) ETD pilot. Thinking is disabled: the model answers with JSON directly.

This release supersedes cometadata/etd-metadata-parsing-qwen3.5-4b-dapo (trained on the first 1,259 annotated records). It is trained on the 2,999-record cometadata/etd-parsing-train-test dataset with an institution-held-out split, and it is a weight-space average ("soup") of three checkpoints (see Recipe).

Results

Test split: 574 documents from 171 degree-granting institutions, none of which appear in training (the dataset's test split). Scoring: per-field normalized matching (case, diacritics and punctuation folded; order-insensitive name lists with token-F1 partial credit above 0.5; strict null semantics), aggregate = unweighted mean of the 10 field scores, exact = fraction of documents with all 10 fields correct. Greedy decoding, max 1,024 tokens.

Paired bootstrap over documents (10k resamples), this model minus baseline, 95% CI:

• vs Claude Opus 5 3-shot: +3.6 points, CI [+2.8, +4.5]
• vs previous release: +1.5 points, CI [+1.0, +2.0]
• vs Qwen3.5-4B zero-shot: +7.3 points, CI [+6.5, +8.0]

Per field (test): title 0.930 | creators 0.928 | advisors 0.888 | date_issued 0.984 | degree_grantor 0.847 | degree_level 0.840 | degree_discipline 0.877 | degree_department 0.761 | language 0.929 | coverage_spatial 0.686

Per language (test, n): en 0.841 (259), fr 0.886 (167), es 0.879 (83), pt 0.925 (58), nl 0.707 (6), de 0.950 (1)

Validation split (138 docs, held-out institutions, used only for checkpoint selection): aggregate 0.897, exact record 0.167.

Where the remaining errors are: coverage_spatial gold mixes null and filled conventions across annotators and often uses the region rather than the printed city; degree_department and degree_level misses are mostly a choice between two printed candidates (college vs department, DOCTORAT vs Docteur); bilingual title pages (e.g. Serbian/English) are labeled from the English page. About 10% of gold values are not present in the extracted text at all.

Recipe

This model is the uniform average of three Qwen3.5-4B checkpoints:

1. cometadata/etd-metadata-parsing-qwen3.5-4b-dapo — the previous release (DAPO on the first 1,259 records; seed 43, step 40).
2. A DAPO run from stock Qwen3.5-4B on the new training split, step 15 (its validation peak), plain reward.
3. A DAPO run from stock Qwen3.5-4B on the new training split with a noise-masked reward (fields whose gold value is not findable in the prompt text are dropped from the reward and the weights renormalized; null golds and language are never masked) and reward weights raised to 1.5 on degree_department and degree_level, step 20.

On the test split the three ingredients score 0.852, 0.859 and 0.863; the average scores 0.867. Continuing the previous release on the new data (warm start) stayed at 0.853, and a prompt with explicit convention rules did not help, so neither is included.

Data. cometadata/etd-parsing-train-test, institution-held-out split (train 2,399 / test 600). From train: 1,852 RL prompts, 138 validation docs from 59 institutions held out of RL, 180 docs reserved for few-shot exemplars; 157 PDFs without a usable text layer, 98 near-empty records and 6 records with no determinable language were excluded. Input = PyMuPDF text of the first pages joined with [PAGE N] markers under a 4,096-token cap (min 3 pages), after NFKC/ligature/accent repair.

RL. DAPO via SkyRL v0.3.0: full-parameter FSDP2 fine-tune, 8 colocated vLLM engines, 128 prompts × 16 samples per step, mini-batch 32 prompts (4 updates/step), AdamW lr 1e-6 with 40 warmup mini-steps, clip 0.2/0.28 (dual-clip 10), token-mean loss, mean-only advantages (no std normalization), dynamic sampling that resamples groups with reward std < 0.005, KL anchor to the init (k3, coef 1e-3), temperature 1.0, max response 1,024 tokens, thinking disabled, truncated responses scored (not masked). Reward = the same per-field scorer as the evaluation (scripts/scoring.py), weighted (title 2, creators 2, advisors 1.5, others 1, language/coverage_spatial 0.5). Checkpoints selected on validation reward; exports every 5 steps. One run ≈ 30 steps × 9 min on 8×H100.

Soup. Uniform weight-space average of the listed checkpoints (all Qwen3.5-4B full models, fp32 average cast back to bf16). Adding a fourth checkpoint (a second base-init seed) did not change the test score.

Prompt

The model was trained and evaluated with exactly this system prompt and a user message holding the page text. Use it verbatim.

You are an expert at reading the front matter of electronic theses and dissertations (ETDs) and extracting their bibliographic metadata. The user will show you the first pages of a thesis or dissertation — title page, repository cover sheet, approval/signature page, etc. — and your job is to extract the metadata fields in a structured format.

Documents are mostly in Portuguese, English, Spanish, French, German, or Dutch. Copy every field value in the document's original language and wording — do NOT translate (e.g. keep 'Doutor em Ciências' as is; do not render it as 'Doctor of Philosophy'; keep 'Doktorgrades' rather than 'Doctorate').

If a field's value is not stated anywhere in the document, emit null (or [] for advisors). Do not guess or infer values that are not printed.

### JSON Schema

{
  "type": "object",
  "properties": {
    "title": {
      "type": [
        "string",
        "null"
      ],
      "description": "Full thesis/dissertation title, exactly as printed."
    },
    "language": {
      "type": [
        "string",
        "null"
      ],
      "description": "ISO 639-1 code of the document language, lowercase (e.g. 'pt', 'en', 'es', 'fr', 'de', 'nl')."
    },
    "creators": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Author name(s), exactly as printed."
    },
    "advisors": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Advisor / supervisor / director name(s), without titles like Prof. or Dr. Empty list if none are stated."
    },
    "degree_discipline": {
      "type": [
        "string",
        "null"
      ],
      "description": "Field of study of the degree (e.g. 'Statistics', 'Engenharia El\u00e9trica')."
    },
    "degree_grantor": {
      "type": [
        "string",
        "null"
      ],
      "description": "Institution granting the degree, exactly as printed."
    },
    "coverage_spatial": {
      "type": [
        "string",
        "null"
      ],
      "description": "City/state/country of the institution as stated in the document (e.g. 'S\u00e3o Paulo', 'Louisville, Kentucky')."
    },
    "degree_department": {
      "type": [
        "string",
        "null"
      ],
      "description": "Department, school, faculty, or program within the institution."
    },
    "degree_level": {
      "type": [
        "string",
        "null"
      ],
      "description": "Degree sought, in the document's own words (e.g. 'Doctor of Philosophy', 'Mestre', 'DOCTORAT', 'Doktorgrades')."
    },
    "date_issued": {
      "type": [
        "string",
        "null"
      ],
      "description": "4-digit year the thesis was issued/defended (e.g. '2009')."
    }
  },
  "required": [
    "title",
    "language",
    "creators",
    "advisors",
    "degree_discipline",
    "degree_grantor",
    "coverage_spatial",
    "degree_department",
    "degree_level",
    "date_issued"
  ],
  "additionalProperties": false
}

### Example Output

{
  "title": "Modelagem numérica de escoamentos turbulentos em canais abertos",
  "language": "pt",
  "creators": [
    "Ana Beatriz Ferreira"
  ],
  "advisors": [
    "Carlos Eduardo Lima"
  ],
  "degree_discipline": "Engenharia Civil",
  "degree_grantor": "Universidade Federal do Paraná",
  "coverage_spatial": "Curitiba",
  "degree_department": "Departamento de Engenharia Civil",
  "degree_level": "Mestre em Engenharia Civil",
  "date_issued": "2015"
}

### Summary

Read the pages carefully, then respond with a single JSON object in the format specified above containing the extracted metadata.

User message format:

[PAGE 1]
<text of page 1>

[PAGE 2]
<text of page 2>
...
[... REMAINING PAGES OMITTED ...]      (only if pages were dropped by the token cap)

Usage

The easiest path is the end-to-end script shipped in this repo under inference/: PDF in, JSON out, with the exact text extraction, cleanup, page budgeting and prompt used in training.

pip install "vllm>=0.23" "transformers>=5" pymupdf
huggingface-cli download cometadata/etd-metadata-parsing-qwen3.5-4b-dapo-v2 --include "inference/*" --local-dir .
python inference/etd_extract.py --pdf-dir /path/to/pdfs --out preds.jsonl

See inference/README.md for options and the output format. Calling the model directly:

from vllm import LLM, SamplingParams
llm = LLM(model="cometadata/etd-metadata-parsing-qwen3.5-4b-dapo-v2", trust_remote_code=True, enable_prefix_caching=False)
msgs = [{"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "[PAGE 1]\n...first pages of the ETD..."}]
out = llm.chat(msgs, SamplingParams(temperature=0, max_tokens=1024,
               stop_token_ids=[248044, 248046]),
               chat_template_kwargs={"enable_thinking": False})
print(out[0].outputs[0].text)   # a single JSON object

This is the full multimodal-architecture checkpoint used text-only; disable prefix caching with vLLM (Qwen3.5 gated-DeltaNet state). Requires vLLM ≥ 0.23 / transformers ≥ 5.

Compute and cost

Trained on one 8×H100 node on SF Compute (24 node-hours, $432) plus evaluation on Stanford's Marlowe cluster. Frontier baselines via OpenRouter (~$21).

Citation

COMET ETD metadata parsing pilot, 2026. Code: https://github.com/cometadata/etd-affiliation-parsing