NexteraBERT-Mezzoforte-220M-en

NexteraBERT is a fast, long-context bidirectional encoder. This repository holds the main model of the NexteraBERT paper: an English model with 212.6M parameters, pretrained with masked language modeling (MLM) on 130B tokens.

The encoder has 18 blocks. Each block has a token mixer and a squared-ReLU MLP, both with pre-normalization and a residual connection. The token mixers are:

• SnowLily (8 blocks): an input-dependent gating liquid mixer derived from liquid time-constant networks. Its cost is linear in input length.
• NexteraSWA (5 blocks): sliding-window attention with a 256-token window.
• NexteraSelfAttention (3 blocks): full attention.
• NexteraHRA (2 blocks): attention over bands of 4 tokens, a cheaper global path.

All attention blocks use grouped-query attention (16 query heads, 8 key/value heads). The full-attention and HRA blocks use SSSMax (Stable Scalable-Softmax), which scales the attention logits with the number of keys so that attention stays selective on long inputs.

Pretrain and modeling code: NexteraBERT Repository

llama cpp style C++ inference code: NexteraBERT.cpp

Results

(a, b) GLUE and MTEB v2 against pretraining tokens: NexteraBERT at 1.3B, 13B, and 130B tokens, with baselines evaluated under the same protocol. (c) H200 NVL throughput with 65,536 tokens per batch. (d) Exponentiated masked-token loss; the dotted line marks 8,192 tokens, the longest training length, and the percentages give the increase from 8,192 to 65,536 tokens.

Comparison with ModernBERT-base

Both models are fine-tuned and evaluated under the same protocol. NexteraBERT uses 130B pretraining tokens and ModernBERT-base 2T. The better score is in bold.

GLUE (validation sets)

Scores are means over 5 seeds (MRPC, STS-B, RTE), 4 (CoLA), 3 (SST-2), or 1 (QQP, MNLI, QNLI). MRPC, STS-B, RTE, and QNLI start from the MNLI-tuned encoder.

MTEB v2 (English)

Both models are fine-tuned with supervised SimCSE on 312,663 NLI triplets (contrastive loss, maximum length 64) and evaluated at maximum length 512.

Retrieval

Both models are fine-tuned on MS MARCO (1.25M triplets with one hard negative, contrastive loss, mean pooling). BEIR datasets are evaluated at maximum length 512, the code tasks and MLDR at 8,192.

BEIR, per dataset (nDCG@10) NanoBEIR, per subset (nDCG@10)

Usage

The model code is included in this repository, so pass trust_remote_code=True. Tested with transformers 5.12 and PyTorch 2.12.

Masked-token prediction

from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline

repo = "RikkaBotan/NexteraBERT-Mezzoforte-220M-en"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)

fill = pipeline("fill-mask", model=model, tokenizer=tokenizer)
print(fill("The capital of France is [MASK]."))

Hidden states

import torch
from transformers import AutoModel, AutoTokenizer

repo = "RikkaBotan/NexteraBERT-Mezzoforte-220M-en"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()

batch = tokenizer(["hello world", "a second sentence"], padding=True, return_tensors="pt")
with torch.no_grad():
    hidden = model(**batch).last_hidden_state          # (batch, length, 1024)

mask = batch["attention_mask"].unsqueeze(-1).to(hidden.dtype)
embedding = (hidden * mask).sum(1) / mask.sum(1)       # mean pooling

This is a pretrained encoder, not a sentence-embedding model. Fine-tune it (for example with contrastive learning) before using it for retrieval or similarity.

Unpadded inference

import torch
from transformers import AutoModel, AutoTokenizer

repo = "RikkaBotan/NexteraBERT-Mezzoforte-220M-en"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, trust_remote_code=True, unpadding=True).cuda().eval()

texts = ["a short query", "a much longer passage about something else entirely"]
batch = tokenizer(texts, padding=True, return_tensors="pt").to("cuda")
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    hidden = model(**batch).last_hidden_state          # (batch, length, 1024)

Fine-tuning for classification

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "RikkaBotan/NexteraBERT-Mezzoforte-220M-en",
    trust_remote_code=True,
    num_labels=3,
)

The classification head is newly initialized and must be trained.

Training

Architecture

Files

• config.json: model configuration, including the auto_map for trust_remote_code
• model.safetensors / pytorch_model.bin: encoder weights
• mlm_head.safetensors: masked-LM prediction head, loaded by AutoModelForMaskedLM
• tokenizer.json, tokenizer_config.json: tokenizer
• modeling_nexterabert_hf.py: transformers model classes
• modeling_nexterabert.py, configuration_nexterabert.py: model and configuration code
• loading.py, export.py, __init__.py: loading and export helpers
• assets/summary.png: results figure

🩵 About me 🩵

Japanese independent researcher having shy and pampered personality. Twin-tail hair is a charm point. Interested in nlp. Usually using python and C.

X(Twitter): https://twitter.com/peony__snow