Lewis-44M-Bert-V3

Lewis-44M-Bert-V3 is a compact (≈ 43.5M parameter) English encoder pre-trained from scratch with ELECTRA-style replaced-token detection on ~2.7B unique tokens (one epoch) of curated multi-domain text. It reuses the modernized architecture of Lewis-44M-Bert (RoPE, RMSNorm, GeGLU, QK-norm) and was trained entirely on a single consumer GPU (RTX 5050, 8 GB VRAM).

GLUE dev average: 80.5 (8 tasks, WNLI excluded), up from 75.2 for the MLM-trained v2 model at the same size (the fine-tuning recipe also changed; see Development history). On the official GLUE test server it scores 76.2 (9 tasks including WNLI; 77.6 over the same 8 tasks as the dev average, see GLUE test-server results). It has also been fine-tuned and evaluated on SQuAD v1.1 (83.8 F1 dev), CoNLL-2003 NER (87.8 test F1), topic and sentiment classification, and measured for inference speed.

Loading requires trust_remote_code=True (custom architecture).

The tokenizer adds the <s> … </s> wrapper (ids 2 and 3) for you, exactly as in pre-training. Only if you tokenize with your own code do you need to add them yourself: skipping them costs more than one GLUE point.

---

Table of contents

1. Model details
2. Tech stack
3. Training data
4. Training procedure
5. Benchmarks
6. Inference speed
7. How to use
8. Why use it
9. Where to use it
10. Limitations and risks
11. Development history
12. Files, citation

---

Model details

Special tokens: <pad>=0, <unk>=1, <s>=2 (BOS), </s>=3 (EOS), <mask>=4. Single sentence: <s> A </s>. Sentence pair: <s> A </s> B </s>. The tokenizer adds these automatically. The sequence-classification head pools position 0, which is why <s> matters.

What was released: only the ELECTRA discriminator, the encoder you fine-tune. The small generator used during pre-training is thrown away, as in the original ELECTRA recipe.

How ELECTRA pre-training works

Instead of masking tokens and asking the model to guess them (BERT/MLM), ELECTRA trains two networks:

1. A small generator (4-layer masked language model) fills randomly masked positions with plausible tokens.
2. The discriminator (the released model) looks at every token and decides: original or replaced?

The discriminator gets a learning signal from all tokens, not only the ~15% that were masked, so it learns more per training token.

---

Tech stack

---

Training data

About 2.68B unique tokens, each seen once. Documents were tokenized with a tokenizer trained on the same mix, packed into 512-token chunks as <s> … </s>, and shuffled globally. 3,875 sequences (about 0.2%) are held out for validation, the same set used by v1 and v2.

The data was collected in two stages: ~1.0B tokens for v1 (FineWeb-Edu 50%, Wikipedia 25%, Gutenberg 15%, Cosmopedia 10%) and 1.68B new tokens for v2 (FineWeb-Edu, Gutenberg, Wikipedia, news, Q&A). Documents already used in the first stage were skipped, and exact duplicates were removed within the new data. V3 trains on both stages together, from scratch, for one epoch.

---

Training procedure

ELECTRA pre-training hyperparameters (V3)

Training diagnostics (step ~27,000): generator MLM loss ≈ 2.3, discriminator loss ≈ 0.19, ≈ 8.3% of tokens replaced, replaced-token precision ≈ 73%. Generator and discriminator stayed balanced throughout, with no divergence.

Compute: about 22 hours for V3 at ≈ 34K tokens/s on one RTX 5050 (8 GB). The whole project (v1 ≈ 21 h, v2 ≈ 10 h, V3 ≈ 22 h) took roughly 53 hours of single-GPU time.

Fine-tuning recipe used for the GLUE numbers

Other settings: AdamW, weight decay 0.01, warmup 6–10%, linear decay, max length 128, bf16, final-epoch score reported (no best-epoch picking), median seed reported. No dropout override was applied (the config default was used); hidden_dropout_prob=0.1 has not been tested.

---

Benchmarks

GLUE (dev set)

All GLUE numbers are dev-set results (8 tasks, WNLI excluded). They are not GLUE test-server scores.

v2 and V3 used different fine-tuning recipes (v2: basic single-run fine-tuning; V3: MNLI transfer, layer-wise LR decay, more epochs, multi-seed medians). The jump therefore combines the model and the recipe. See Development history for the part due to ELECTRA alone.

Under the ELECTRA paper's stricter convention (MCC for CoLA, Spearman for STS-B, accuracy elsewhere, MNLI-matched) the average is 80.45.

Seed variance (final-epoch score, 5 seeds):

MNLI, QQP, QNLI and SST-2 are single runs. Expect about ±0.2–0.3 on rerun (SST-2 can move more: its dev set has only 872 examples).

GLUE test-server results

Submitted to the official GLUE evaluation server (test labels are hidden). GLUE score: 76.2 (average over 9 tasks, MNLI counted as the mean of matched and mismatched).

• Averages: 76.2 over all 9 tasks. Over the same 8 tasks as the dev average (WNLI excluded, MNLI as the mean of matched and mismatched, and MRPC, STS-B and QQP as the mean of their two metrics) the test average is 77.6, against 80.5 on dev.
• WNLI was not trained. The submission predicts the majority class of the training set for every example, so it is not a model result.
• Largest dev-to-test gaps: RTE (−5.3, a very small test set), CoLA (−3.7, one of the two tasks whose settings were tuned on dev), and QQP F1 (69.7 against 86.9). QQP's test F1 is typically far below its dev F1 because the two sets are not balanced the same way, so QQP accuracy (88.4 against 90.3) is the more comparable number.
• How the predictions were produced: for RTE, MRPC, STS-B and CoLA (10, 10, 5 and 8 seeds) the test predictions come from the median-dev seed; SST-2, QNLI and QQP are single runs; MNLI and AX come from the MNLI model; RTE, MRPC and STS-B start from that MNLI model. The per-task learning rate, layer-wise decay and epochs were those in finetune_glue_v2.py, whose dev scores were within about one point of the dev table above.
• Single submission, no repeated leaderboard attempts.

Question answering: SQuAD v1.1 (dev set)

Setup: a linear start/end span head on the encoder output, input <s> question </s> context </s>, max length 384 with a sliding window (stride 64) for long contexts, questions capped at 300 characters, AdamW (weight decay 0.01), LR 1e-4, 2 epochs, batch 32, 6% warmup then linear decay, bf16, standard best-span decoding (top 20 start/end candidates, answers up to 30 tokens). Single run, single seed, dev set only. The learning rate was not tuned.

Named-entity recognition: CoNLL-2003

Validation F1 by epoch: 90.85, 92.41, 92.51, 92.60. Setup: linear token-classification head, label on the first sub-token of each word, max length 256, AdamW (weight decay 0.01), LR 1e-4, 4 epochs, batch 32, 6% warmup then linear decay, bf16, entity-level F1 with seqeval. Single run, single seed. The final epoch was used (no best-epoch picking), and the test set was evaluated once. The test split of CoNLL-2003 is known to be harder than the validation split, which explains most of the gap between the two scores.

Other classification tasks (10k training examples)

Setup: 10,000 randomly sampled training examples, the model's standard sequence-classification head, LR 5e-5, 3 epochs, batch 32, 6% warmup then linear decay, AdamW (weight decay 0.01), final-epoch scores, single seed (42). IMDB is truncated to 256 tokens, which limits accuracy on long reviews.

Low-data fine-tuning

Accuracy (mean ± std over 3 seeds) when fine-tuning on only n labeled examples. Same recipe for every size: LR 1e-4, batch 32, max length 128, 10 epochs for n ≤ 500, 5 epochs for n = 1,000, 3 epochs for n = 5,000.

Reading this fairly: topic classification (AG News) is learned quickly from a few hundred examples. Sentiment (SST-2) needs more data, with the model near chance at 100 examples and in the low 80s at 5,000 (88.6 with the full training set). The learning rate and epoch counts were not tuned per size, so the smallest sizes may be under-trained.

What these results do not show

• The GLUE test-server result comes from a single submission, with the median-dev seed chosen for the multi-seed tasks, and WNLI was not trained.
• SQuAD, NER and the low-data runs are single-recipe (one learning rate) and, except where stated, single-seed.
• No retrieval, sentence-embedding or token-level tasks beyond CoNLL-2003 NER.
• CoLA and SST-2 settings were chosen after looking at earlier dev runs, so those two tasks are slightly tuned on dev.
• MNLI, QQP, QNLI and SST-2 are single-seed.
• Learning-rate sensitivity has not been measured; the numbers above come from the specific learning rates listed.

---

Inference speed

Encoder forward pass only (no tokenization), random token inputs, 30 timed iterations after warmup. GPU: NVIDIA RTX 5050 laptop GPU with bf16 autocast. CPU: laptop CPU, fp32, default PyTorch thread count. Peak memory is the GPU allocation during inference.

• A single 128-token sentence takes about 17 ms on the GPU (limited by kernel-launch overhead at this size).
• Batched GPU throughput reaches about 105K tokens/s (roughly 800 sentences/s at 128 tokens).
• Inference peak memory stays under 600 MB even at batch 32 and 512 tokens.
• CPU timings vary with background load and clock speed: batch-1, 128-token latency ranged from about 50 ms to 85 ms across runs on the same machine. Treat CPU numbers as indicative.

---

How to use

Install

pip install torch transformers safetensors

Classification (the model has no MLM head)

Only the ELECTRA discriminator is released, so use it as an encoder. The classification head is new and randomly initialized: fine-tune before trusting its outputs.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "vprojectx/Lewis-44M-Bert-V3"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(
    repo, trust_remote_code=True, num_labels=2
).eval()

# The tokenizer adds <s> ... </s> (and <s> A </s> B </s> for pairs) automatically.
batch = tok(["A great, funny movie.", "Terrible plot."],
            padding=True, truncation=True, max_length=128, return_tensors="pt")
with torch.no_grad():
    logits = model(**batch).logits

# sentence pairs (NLI, paraphrase, ...):  tok(premises, hypotheses, padding=True, ...)

If you use your own tokenization code, make sure every sequence starts with id 2 (<s>) and ends with id 3 (</s>), with pairs as <s> A </s> B </s>.

Sentence features (and a backbone for span / token tasks)

with torch.no_grad():
    hidden = model.Lewis_44M_Bert(
        batch["input_ids"], attention_mask=batch["attention_mask"], return_dict=True
    ).last_hidden_state
m = batch["attention_mask"].unsqueeze(-1).float()
embedding = (hidden * m).sum(1) / m.sum(1)      # mean pooling

hidden is the per-token output you can put a linear head on for question answering (start/end logits) or token classification (NER), which is how the SQuAD and CoNLL-2003 results above were produced.

This encoder was not trained with a sentence-similarity objective. For retrieval or semantic search, fine-tune it on pairs first.

Recommended fine-tuning settings

General tips from the GLUE work:

1. Layer-wise LR decay of 0.8, top LR about 1.5e-4–3e-4 (5e-5 with no decay worked best on CoLA and SST-2).
2. For small tasks (RTE, MRPC, STS-B), fine-tune on MNLI first and start from that checkpoint.
3. 10 epochs on small tasks, 3 on large ones, batch 16–32.
4. Several seeds on small datasets; report the median.
5. Always keep the <s> … </s> wrapper (the tokenizer does it for you).

Reproduce the GLUE evaluation

pip install torch transformers datasets scikit-learn scipy tqdm huggingface_hub
hf download vprojectx/Lewis-44M-Bert-V3 finetune_glue_final.py --local-dir .
python finetune_glue_final.py --out ./glue_final

The script downloads this model, trains all 8 tasks (MNLI first), writes results.json and seed_stats.json, and prints the GLUE average. It resumes if interrupted; the full run takes roughly 3–4 hours on a laptop-class GPU.

---

Why use it

• Small and cheap. 44M parameters. It fine-tunes on an 8 GB laptop GPU, runs on CPU, and needs under 600 MB of GPU memory for inference at batch 32 and 512 tokens.
• Covers the main task types. Sentence classification and inference (GLUE 80.5 dev, 77.6 test over 8 tasks), span extraction (SQuAD v1.1 F1 83.8) and token-level tagging (CoNLL-2003 NER F1 87.8 test).
• Fast. About 17 ms per sentence on a laptop GPU and about 105K tokens/s batched.
• Efficient pre-training. ≈ 2.7B tokens in one epoch, about 22 hours on one consumer GPU. Easy to extend, retrain or reproduce.
• Handles small, noisy datasets well such as RTE (72.6) and CoLA (52.3), where replaced-token detection helps.
• Modern architecture. RoPE, RMSNorm, GeGLU and QK-norm, which tend to improve stability and quality per parameter.
• Curated, open data. Educational-quality web text plus encyclopedic, literary, news, Q&A and synthetic sources; no distillation from a teacher model.
• Documented recipe. Training settings, fine-tuning settings, seed spread and the evaluation script are all published.

Where to use it

Good fits:

• Text classification (topic, intent, emotion, spam, toxicity after fine-tuning)
• Natural language inference and entailment checks
• Paraphrase and duplicate-question detection
• Semantic textual similarity (after fine-tuning)
• Extractive question answering and named-entity recognition
• Low-latency, on-device or CPU inference; budget-limited APIs
• A compact base for domain adaptation (legal, medical, support tickets) with a small labeled set
• Teaching, ablations and research baselines at small scale

Not a good fit:

• Text generation, chat, instruction following (encoder only)
• Documents longer than 512 tokens
• Non-English text
• Sentiment tasks with only a few hundred labeled examples (see the low-data results)
• Tasks where the last few points of accuracy matter: validate on your own data first
• High-stakes decisions without task-specific validation and human review

---

Limitations and risks

• Dev numbers are optimistic. The GLUE dev average (80.5) is higher than the GLUE test-server result (77.6 over 8 tasks, 76.2 over 9 including an untrained WNLI). SQuAD is dev-only. NER is reported on both validation and test.
• Light tuning on dev. The CoLA and SST-2 recipes were chosen after earlier dev runs; CoLA is 3.7 points lower on test than on dev.
• Single-seed results for MNLI, QQP, QNLI, SST-2, SQuAD, NER and the 10k-example classification runs. Differences of a few tenths of a point are within run-to-run noise.
• One learning rate per task for SQuAD, NER and low-data runs; sensitivity to the learning rate has not been measured.
• Sentiment with very little data is weak: SST-2 accuracy is 58% at 100 examples and 71% at 1,000.
• English only, cased 30k vocabulary, maximum 512 tokens. SQuAD contexts longer than the window are handled with a sliding window.
• CPU speed numbers are noisy and depend on the machine.
• Packed training sequences let attention cross document boundaries.
• Validation chunks come from the same source documents as training, so MLM-style perplexities from earlier stages are slightly optimistic.
• Not evaluated for bias, toxicity or factual reliability; it inherits the biases of web, encyclopedic, news and 19th/20th-century literary text.
• No evaluation yet on retrieval or sentence-embedding benchmarks.
• Custom code: loading needs trust_remote_code=True. Review the .py files in the repo before running them.

---

Development history

---

Files, citation

Files: config.json, model.safetensors, configuration_Lewis_44M_Bert.py, modeling_Lewis_44M_Bert.py, tokenizer.json, tokenizer_config.json, special_tokens_map.json, plus (recommended) finetune_glue_final.py, results.json, seed_stats.json.

@misc{lewis44mbertv3,
  title  = {Lewis-44M-Bert-V3: a compact ELECTRA-style encoder pretrained from scratch on a single consumer GPU},
  author = {Viraj Salunke},
  year   = {2026},
  url    = {https://huggingface.co/vprojectx/Lewis-44M-Bert-V3}
}

References

• Clark et al., ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, ICLR 2020. https://arxiv.org/abs/2003.10555
• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019.
• Wang et al., GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, ICLR 2019.
• Rajpurkar et al., SQuAD: 100,000+ Questions for Machine Comprehension of Text, EMNLP 2016.
• Tjong Kim Sang and De Meulder, Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition, 2003.