much1na/ettin-400m-squadv2

This model is fine-tuned from jhu-clsp/ettin-encoder-400m for extractive question answering. Given a question and a passage, it marks the span of the passage that answers the question. It was trained on the SQuAD 2.0 train split of rajpurkar/squad_v2.

Data

• Training: SQuAD 2.0 train, 130319 questions.
• Evaluation: SQuAD 2.0 validation, 11873 questions.

Hyperparameters

learning_rate = 5e-05
batch_size = 32 (global, over 1 process(es); 8 per GPU x 4 accumulation steps)
optimizer = adamw_torch
gradient_checkpointing = True
n_epochs = 2
max_steps = -1 (-1 means use n_epochs)
max_seq_len = 384
doc_stride = 128
warmup_ratio = 0.1
weight_decay = 0.01
seed = 42
precision = fp16

Results

Usage (transformers 5; the question-answering pipeline is not available there)

import torch
from transformers import AutoModelForQuestionAnswering, AutoTokenizer

tok = AutoTokenizer.from_pretrained("much1na/ettin-400m-squadv2")
model = AutoModelForQuestionAnswering.from_pretrained("much1na/ettin-400m-squadv2")
question = "Who designed the Eiffel Tower?"
context = "The Eiffel Tower was built by the company of Gustave Eiffel."
enc = tok(question, context, return_tensors="pt", truncation="only_second", max_length=384, return_offsets_mapping=True)
offsets = enc.pop("offset_mapping")[0].tolist()
ctx = [i for i, s in enumerate(enc.sequence_ids(0)) if s == 1]
with torch.no_grad():
    out = model(**enc)
start = ctx[int(out.start_logits[0, ctx].argmax())]
end = ctx[int(out.end_logits[0, ctx].argmax())]
print(context[offsets[start][0]:offsets[end][1]])

Trained with train.py (v2 settings, 1 process(es), fp16) on 2026-11-08.