📜 led-base-16384-scrolls-qasper

A 161.84M-parameter Longformer Encoder-Decoder (LED) fine-tuned on SCROLLS QASper for full-paper scientific Question Answering.

A Longformer Encoder-Decoder (allenai/led-base-16384) Seq2Seq model fine-tuned on SCROLLS QASper across 16,384 tokens context, achieving 35.30% Held-Out Test SCROLLS F1 on 1,399 unique questions (surpassing the 24.2% LED-base baseline reported in the SCROLLS paper).

---

✨ Highlights

• 🔭 99.86% Full-Paper Context Coverage (16,384 Tokens): 99.86% of research papers fit completely within the context window without truncating Methods, Experiments, or Discussion sections.
• 🎯 State-of-the-Art Test Generalization (35.30% Test F1): Evaluated using official SCROLLS multi-reference evaluation (4-Beam Search) across all 1,399 test questions:
  • Free-Form / Extractive (n = 1,061): 28.30% F1
  • Yes / No (n = 213): 64.42% F1
  • Unanswerable (n = 125): 45.08% F1
• ⚡ Ampere Compute & Memory Engineering: Native bfloat16 single-shard checkpoint (model.safetensors, 308 MB), Dynamic Token-Budgeted Batching (24,576 tokens/batch), and Gradient Checkpointing capping peak VRAM at 10.25 GB / 39.49 GB on 1× NVIDIA A100.

---

🚀 Quick Start

Run long-context scientific QA up to 16,384 tokens with 🤗 Transformers (assign global_attention_mask = 1 to <s> and all question tokens before \n\n, matching training):

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

repo = "GiangSon-5/led-base-16384-scrolls-qasper"
tokenizer = AutoTokenizer.from_pretrained(repo)
dtype = torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float32
model = AutoModelForSeq2SeqLM.from_pretrained(repo, dtype=dtype)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device).eval()

question = "What attention mechanism allows Longformer to process long documents linearly?"
paper_context = """
Transformer-based models suffer from quadratic O(N^2) self-attention complexity. To address this,
Longformer introduces a sliding window local attention pattern combined with global attention on
selected tokens, achieving linear O(N) scaling across 16,384 tokens.
"""

prompt = f"{question}\n\n{paper_context.strip()}"
sep_idx = prompt.find("\n\n")

enc = tokenizer(prompt, max_length=16384, truncation=True, return_offsets_mapping=True, return_tensors="pt")
offsets = enc.pop("offset_mapping")[0].tolist()

# Set global attention on <s> and question tokens
global_attention_mask = torch.zeros_like(enc["input_ids"])
global_attention_mask[0, 0] = 1
for idx, (off_s, off_e) in enumerate(offsets[1:], start=1):
    if off_s < off_e and off_s < sep_idx:
        global_attention_mask[0, idx] = 1

with torch.no_grad():
    generated_ids = model.generate(
        input_ids=enc["input_ids"].to(device),
        attention_mask=enc["attention_mask"].to(device),
        global_attention_mask=global_attention_mask.to(device),
        max_new_tokens=128,
        num_beams=4,
        no_repeat_ngram_size=3,
        length_penalty=1.0,
        early_stopping=True,
    )

print("Answer:", tokenizer.decode(generated_ids[0], skip_special_tokens=True).strip())

---

💬 Sample Predictions

Empirical predictions from the standalone reloaded bfloat16 checkpoint across representative Held-Out Test questions (4-Beam Search):

---

1. Architecture Specs

Extracted verbatim from serialized config.json and tokenizer_config.json:

---

2. Training Data & Procedure

Data Engineering

• Dataset: tau/scrolls (qasper), covering 2,567 train pairs (2,490 unique IDs), 984 validation, and 1,399 held-out test unique questions.
• Strict Split Isolation (Zero Contamination): Strict partition integrity with zero question ID leakage across subsets (Train ∩ Val ∩ Test = ∅) and exact prompt deduplication.
• Offset-Aligned Global Masking: Uses return_offsets_mapping=True to accurately align the \n\n boundary with BPE tokens, anchoring bidirectional global attention to the question.
• Tensor Alignment & Caching: All inputs are padded to multiples of the sliding window (pad_to_multiple_of = 1024) for optimal Ampere Tensor Core execution, with in-memory caching across shared paper contexts.

Training Configuration

---

3. Evaluation

Evaluated using official SCROLLS multi-reference scoring (metric_max_over_ground_truths) with 4-Beam Search:

Official SCROLLS Paper (EMNLP 2022) LED-base reference baseline: 24.2% F1.

---

4. Limitations & Ethical Considerations

• Domain Scope: Fine-tuned exclusively on English NLP/CS scientific research articles from tau/scrolls (qasper). Performance on non-scientific, conversational, clinical, legal, or multilingual documents is not evaluated.
• Verification Required: At 35.30% F1 (19.94% Exact Match), generated outputs should assist research literature review rather than replace authoritative human expert verification.
• Hallucination Risk: While seq2seq cross-attention grounds answers in provided papers, unanswerable questions with low confidence may occasionally yield plausible-sounding extractions.

---

5. Google Colab & Reproducibility

---

6. Citation & References

• 📄 Longformer & LED: Beltagy et al., 2020. Longformer: The Long-Document Transformer. arXiv:2004.05150
• 📄 QASper Dataset: Dasigi et al., 2021. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. arXiv:2105.03011
• 📄 SCROLLS Benchmark: Shaham et al., 2022. SCROLLS: Standardized CompaRison Over Long Language Sequences. arXiv:2201.03533

---

7. License & Author

• License: Released under the Apache-2.0 License (inherited from upstream allenai/led-base-16384).
• Author: Giang Sơn (GiangSon-5).