NexteraBERT-Mezzoforte-220M-en-gguf

Quantized GGUF files of RikkaBotan/NexteraBERT-Mezzoforte-220M-en for CPU inference with NexteraBERT.cpp, a ggml-based runtime for the NexteraBERT architecture.

Each file is self-contained: it holds the weights, the hyperparameters, the masked-LM head, and the tokenizer. The files use the GGUF container and ggml's quantization formats, but llama.cpp does not know this architecture, so they load in NexteraBERT.cpp and not in llama-server or llama-embedding.

Pretrain and modeling code: NexteraBERT Repository

Results

(a) Masked-LM accuracy of each GGUF file type on 35,500 masked positions; the dotted line is the PyTorch fp32 model and the bars give the standard error of the difference from it. (b) CPU latency of one sequence on a Core Ultra 7 265K: NexteraBERT as GGUF Q4_K_M against LFM2.5-Encoder, ModernBERT-base, and BERT-base in PyTorch fp32. The figures at the right give the speed-up at 1,024 tokens.

Accuracy against the fp32 model

Every file is compared with the PyTorch fp32 model on the same inputs. Token cosine is the cosine similarity of each token's last_hidden_state (60 documents of 16 to 2,048 tokens) and sentence cosine that of the mean-pooled embedding (2,758 STS-B sentences). MLM accuracy uses 35,500 positions masked at random in 1,000 documents; top-1 agreement is the share of them where the file predicts the same token as fp32. STS-B is the Spearman correlation of the raw embeddings with the gold scores. An unquantized f32 GGUF reproduces PyTorch to float rounding (relative error 1.3e-06).

CPU speed against other encoders

Latency of one sequence in milliseconds (median) on an Intel Core Ultra 7 265K. The PyTorch models run in fp32 with their best of 8, 16, and 20 threads; the GGUF runtime uses its default 18 threads. The fastest is in bold. All quantized file types run at the same speed (Q8_0 / Q5_K_M / Q4_0: 16.0 / 15.7 / 15.9 ms at 128 tokens).

NexteraBERT as GGUF is 1.6× to 3.1× faster than the three PyTorch models. This compares a quantized runtime with unquantized PyTorch on the same CPU; it does not rank the architectures. BERT-base's position table (512) is tiled to reach 1,024 tokens.

Long inputs

The runtime never builds a T × T attention matrix, so 65,536 tokens run in 1.7 GiB of memory. PyTorch on CPU would need about 512 GiB of attention scores at that length and was not run. NexteraBERT was trained on inputs up to 8,192 tokens; the last column shows that the length executes, not the output quality there. The PyTorch times are also not the model's reference speed: its sliding-window blocks use FlexAttention only on CUDA and fall back to dense attention on CPU.

Usage

Build NexteraBERT.cpp and download a file from this repository. The Python binding needs numpy and tokenizers; PyTorch is not required.

git clone https://github.com/Rikka-Botan/NexteraBERT.cpp
cd NexteraBERT.cpp
cmake -B build
cmake --build build --config Release -j
pip install numpy tokenizers huggingface_hub
hf download RikkaBotan/NexteraBERT-Mezzoforte-220M-en-gguf \
    nexterabert-mezzoforte-220m-en-Q4_K_M.gguf --local-dir models

Sentence embeddings

import sys; sys.path.insert(0, "python")
from nexterabert_cpp import NexteraBERT

model = NexteraBERT("models/nexterabert-mezzoforte-220m-en-Q4_K_M.gguf")
embeddings = model.encode(["hello world", "a second sentence"])   # (2, 1024), L2-normalized

encode mean-pools the last hidden states. This is a pretrained encoder, not a sentence-embedding model: the embeddings are those of the base model without fine-tuning.

Masked-token prediction

print(model.fill_mask("The capital of France is [MASK].", top_k=3))

Hidden states

ids = model.tokenize(["a short query", "a much longer passage about something else entirely"])
pooled, hidden = model.encode_ids(ids, return_hidden=True)   # hidden[i]: (length_i, 1024)

Sequences of different lengths share a forward pass without padding, and each result is the same as if the sequence were run alone.

File types

The _M files keep at Q6_K the tensors whose 4-bit error reaches the output most directly: every projection that writes into the residual stream (the mixers' and the MLP's c_proj), the attention value and gate projections, and the MLM head's dense layer. The choice was measured, not copied from llama.cpp.

Files

• nexterabert-mezzoforte-220m-en-Q8_0.gguf: Q8_0 weights, MLM head, and tokenizer
• nexterabert-mezzoforte-220m-en-Q6_K.gguf: Q6_K weights, MLM head, and tokenizer
• nexterabert-mezzoforte-220m-en-Q5_K_M.gguf: Q5_K_M weights, MLM head, and tokenizer
• nexterabert-mezzoforte-220m-en-Q4_K_M.gguf: Q4_K_M weights, MLM head, and tokenizer
• nexterabert-mezzoforte-220m-en-Q4_K_S.gguf: Q4_K_S weights, MLM head, and tokenizer
• nexterabert-mezzoforte-220m-en-Q4_0.gguf: Q4_0 weights, MLM head, and tokenizer
• assets/summary.png: results figure

🩵 About me 🩵

Japanese independent researcher having shy and pampered personality. Twin-tail hair is a charm point. Interested in nlp. Usually using python and C.

X(Twitter): https://twitter.com/peony__snow