BertBM-100m 🚀 fno+gnp+gnn+aegnn+riemann

Transformers only look at tokens. BertBM-100m looks at structure.

BertBM-100m bolts a whole geometry lab onto ModernBERT: Fourier transforms for the rhythm of language, graph networks for who-talks-to-whom, an event-driven graph that skips the parts that didn't change, and a Poincaré manifold where the latents actually live on a curved surface.

FNO | GeoGNP | GNN | AEGNN | Riemannian

---

⚡ Highlights

🧬 The operator stack

🌊 FNO — Fourier Neural Operator

Spectral mixing in the Fourier domain. Your text has rhythm, periodicity and repetition; FNO gives the model a Fourier lens instead of hoping attention figures it out on its own.

🕸️ GeoGNP — Geometric Neural Process

A variational graph operator over the token field. Think of it as a latent map of the sequence: who talks to whom, and how confident are we about that structure.

🕸️ GNN — Learnable Message Passing

Deterministic graph layers with learned gates and (optional) GAT-style edge attention. The graph learns how to propagate instead of trusting a fixed adjacency.

⚡ AEGNN — Asynchronous Event-Driven GNN

The speedrun build. Nodes that didn't change don't get recomputed. Fire an event, update that neighbourhood, leave the rest of the graph alone. Less compute, same brain.

🌀 Riemannian — Poincaré Tangent Mechanism

Latent states live on a hyperbolic manifold. Mixing happens in the tangent space (fast, stable Euclidean updates) and gets mapped back geodesically. Curvature regularization keeps the embeddings from collapsing.

🚦 GateFuse — the router

Every operator is a specialist; GateFuse is the editor. It learns, per sequence, how much of the language field, the spectral view, the geometric view, the learned graph and the event-driven graph to keep. It starts out almost fully on the plain language field (so the pretrained backbone is never stomped on) and the operators earn their way in during training.

🏋️ Training

📈 Results

Learning curve:

📌 Fair-play note: these numbers come from a limited-budget run (see the table above) — they're not directly comparable with production-scale numbers for ModernBERT, which see orders of magnitude more tokens. Use the curve above to compare operator configurations trained under the same protocol.

🔌 Usage

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

repo = "scairesearch/bertbm-100m"
tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)

top = out.logits[0, inputs["input_ids"].shape[1] - 1].topk(5)
for idx, score in zip(top.indices, top.values):
    print(tokenizer.decode([idx]), f"{float(score):.2f}")

Just want the backbone?

The export also ships a plain ModernBERT-compatible checkpoint (model.safetensors.backbone) with the operators removed — drop it in as a modernbert model.

🧪 Try the other configs

The same repo family ships the isolated operator ablations, so you can see what each piece is actually worth:

📝 Citation

This model is part of the BertBM research line:

@misc{bertbm2026,
  title        = {BertBM: Spectral-Geometric Operator Fusion for Transformer Language Models},
  author       = {Niranjan Hasabnis and SCAI Research Labs},
  year         = {2026},
  eprint       = {arXiv:XXXX.XXXXX},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG}
}

📜 License

Apache 2.0