BertBM 🚀 gnn

Transformers only look at tokens. BertBM looks at structure.

BertBM bolts a whole geometry lab onto ModernBERT: Fourier transforms for the rhythm of language, graph networks for who-talks-to-whom, an event-driven graph that skips the parts that didn't change, and a Poincaré manifold where the latents actually live on a curved surface.

GNN

---

⚡ Highlights

🧬 The operator stack

🕸️ GNN — Learnable Message Passing

Deterministic graph layers with learned gates and (optional) GAT-style edge attention. The graph learns how to propagate instead of trusting a fixed adjacency.

🚦 GateFuse — the router

Every operator is a specialist; GateFuse is the editor. It learns, per sequence, how much of the language field, the spectral view, the geometric view, the learned graph and the event-driven graph to keep. It starts out almost fully on the plain language field (so the pretrained backbone is never stomped on) and the operators earn their way in during training.

🏋️ Training

📈 Results

Learning curve:

📌 Fair-play note: these numbers come from a limited-budget run (see the table above) — they're not directly comparable with production-scale numbers for ModernBERT, which see orders of magnitude more tokens. Use the curve above to compare operator configurations trained under the same protocol.

🔌 Usage

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "scairesearch/bertbm-None"
tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)

top = out.logits[0, inputs["input_ids"].shape[1] - 1].topk(5)
for idx, score in zip(top.indices, top.values):
    print(tokenizer.decode([idx]), f"{float(score):.2f}")

Just want the backbone?

The export also ships a plain ModernBERT-compatible checkpoint (model.safetensors.backbone) with the operators removed — drop it in as a modernbert model.

🧪 Try the other configs

The same repo family ships the isolated operator ablations, so you can see what each piece is actually worth:

📝 Citation

This model is part of the BertBM research line:

@misc{bertbm2026,
  title        = {BertBM: Spectral-Geometric Operator Fusion for Transformer Language Models},
  author       = {Niranjan Hasabnis and SCAI Research Labs},
  year         = {2026},
  eprint       = {arXiv:XXXX.XXXXX},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG}
}

📜 License

Apache 2.0