minBERT QQP cross-encoder with swap augmentation and R-Drop

A BERT-base paraphrase detector for Quora Question Pairs, trained from scratch implementation of BERT (minBERT: attention, transformer block, embeddings and AdamW written by hand) for the Deep Learning for Natural Language Processing course at the University of Göttingen, summer semester 2026.

This is the best single model of the QQP study. It is not the model behind the submitted predictions — those come from a five-checkpoint ensemble that scores 0.898. See Relationship to the submitted system.

What it does

Given two questions, it predicts whether they ask the same thing. The pair is encoded jointly as [CLS] q1 [SEP] q2 [SEP] with real segment ids, so every layer of self-attention can attend across the pair. The Part 1 baseline encoded each question separately and concatenated the pooled vectors, which makes the logit additive in the two questions and unable to represent token-level correspondence; replacing it with this cross-encoder is worth +0.111 accuracy, more than every other intervention in the study combined.

Two things are layered on top:

• Swap augmentation. QQP labels are symmetric, but nothing in a cross-encoder enforces that. The baseline changes its prediction on 4.6% of pairs when the two questions are exchanged. Adding (q2, q1, y) for every training pair roughly halves that.
• R-Drop (Liang et al., 2021), α = 0.5. Each batch goes through the model twice with independent dropout masks and the loss adds the symmetric KL between the two predictions. It costs nothing at inference.

Results

Development split of the course QQP data, 33,783 pairs, at a 0.5 threshold.

In-domain accuracy and adversarial robustness move independently. This model has the best development accuracy and the lowest flip rate, and it is near chance on PAWS-Wiki (0.477), whose pairs have high lexical overlap and adversarial labels. A sibling model trained with 20% PAWS replay reaches 0.918 there at unchanged QQP accuracy. If you need robustness to adversarial overlap, this is not the checkpoint to use.

Training

No hyperparameter search was run. The recipe was fixed once from published fine-tuning practice, and every experiment changes exactly one thing against a seed-matched control.

Usage

The architecture is a course implementation, not transformers, so the weights need that code to run. model.safetensors holds the state dict and config.json records the architecture and the training recipe.

import json
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download

repo = "MahmoudMohamed/minbert-qqp-crossencoder-rdrop"
weights = load_file(hf_hub_download(repo, "model.safetensors"))
config = json.load(open(hf_hub_download(repo, "config.json")))

# With the project's multitask_classifier.py on the path:
from types import SimpleNamespace
from multitask_classifier import MultitaskBERT

model = MultitaskBERT(SimpleNamespace(**config["model_config"]))
model.load_state_dict(weights)
model.eval()

Encode a pair as [CLS] q1 [SEP] q2 [SEP] with segment ids 0 and 1, then call model.predict_paraphrase(input_ids, attention_mask, None, None, token_type_ids) and apply a sigmoid. Passing the two questions separately, or dropping the segment ids, will silently give worse results — this is a cross-encoder.

Relationship to the submitted system

The predictions submitted for the course come from an equal-weight probability ensemble of five checkpoints with one tuned decision threshold, selected by five-fold cross-fitting on the development split so that no example helps choose the rule that predicts it. That estimate is 0.898, a gain of 0.006 over this model with a paired bootstrap 95% interval of [0.004, 0.008].

This repository holds one member of that pool — the strongest individual model. Publishing the ensemble would require all five checkpoints, the member list and the threshold.

Limitations

• Not robust to adversarial lexical overlap. 0.477 on PAWS-Wiki is near chance. It relies on word overlap more than a paraphrase model should.
• English questions only, and the domain is Quora questions. Behaviour on other sentence pairs is untested.
• Single seed. The +0.003 R-Drop gain over swap augmentation alone sits close to the seed noise floor of ±0.001–0.002 measured on four configurations across three seeds. It is suggestive, not settled.
• Dev-split numbers. Every figure here is the development split. Official test labels were never read, so no test-set score is claimed.
• Trained on Quora questions, which carry the biases of that platform. Not suitable for any decision about a person.

Provenance and reproduction

Course project for DNLP SS26, University of Göttingen. The training code, diagnostics and the full report — including the negative results that did not survive their controls — live in the group repository. Every number above is recomputed from a tracked diagnostic file by an automated test that fails if a reported value disagrees with its source.

Citation

@misc{abdellahi2026minbertqqp,
  title  = {minBERT QQP cross-encoder with swap augmentation and R-Drop},
  author = {Mahmoud Abdellahi},
  year   = {2026},
  note   = {Deep Learning for Natural Language Processing, University of Göttingen},
  howpublished = {\url{https://huggingface.co/MahmoudMohamed/minbert-qqp-crossencoder-rdrop}}
}

References. BERT · R-Drop · PAWS

The minBERT scaffold derives from Stanford CS224N and CMU 11-711, and parts of the code from Hugging Face Transformers (Apache 2.0).