captionbert-8192-v2

A 58.3M standalone sentence encoder distilled from the geometric consensus of five BERT-family teachers. No expert models at inference: tokenizer + this model, 768-d L2-normalized output.

12 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. 0.53x bert-base.

📄 Technical companion: TECHNICAL.md — the full training recipe, per-task tables, geometry gauges, adapter-library numerics, and the laws this family earned. Week-in-review article: Geometric Memory FT5 — Agreement, Anchors, Addresses.

from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

emb = model.encode(["a cat on a windowsill", "a feline by the window"])   # (2, 768)
(emb[0] @ emb[1]).item()

Benchmark

All ten models measured in one harness, same eight tasks, mean-pooled and L2-normalized, no task tuning. Spearman correlation; mean is the unweighted average over the eight.

all-MiniLM-L6-v2 was contrastively trained on 1B+ curated sentence pairs. It is listed for scale, not as a peer -- nothing here saw a similarity label.

The trunk beats every teacher it was distilled from, and the best of them (distilbert, .5883) by +.0194 -- at 13% of their combined 461M parameters, having never seen a similarity label. The margin comes mostly from STS12, where every teacher collapses to .31-.43 and the trunk holds .50.

With arms it clears the best teacher by +.14 and closes to within .063 of a model trained on a billion curated pairs.

Mean-pooled BERT-family encoders are known-weak sentence encoders -- that is the reason Sentence-BERT exists -- so beating them is an efficiency result rather than a state-of-the-art one. The MiniLM row is in the table to keep that honest.

Geometry

self_cos is the isotropy gauge: the mean cosine between unrelated sentences. Mean-pooled BERT-family embeddings sit in a narrow cone (+.61 to +.96), where cosine cannot discriminate. erank is the participation ratio -- how many of the 768 directions carry variance.

Both track capability almost perfectly across all ten models, and isotropy is the mechanism: no isotropy objective appears anywhere in the training stack. The arms then lift erank 36.6 -> 57.6, the first evidence in this line that adaptation adds usable directions rather than only rotating them.

With AMOE arms (amoe/)

The trunk is frozen; each anchor is 1.6M params across 12 sites. Anchors toggle bit-exact -- all disabled reproduces the bare trunk exactly -- so one artifact serves both the unsupervised baseline and the adapted model.

Best measured roster: equiv + simplify + paraphrase under a trained dispatch, across the full 8-task STS suite:

Eight for eight over every single member, +.1210 mean over the trunk, from 1.6M x 3 adapter params on a model that never moves. Note the BIOSSES column: the strongest member alone reads .5845, below the bare trunk -- the mixture turns a liability into a gain.

Solo anchor scores (trained alone, not masked inside a dispatch):

How many arms, and which

Arms do not add capacity -- they split a roughly conserved amplitude budget. w_k/z = sinh(u_k) / SUM_j cosh(u_j) sums over all anchors, so measured sums run .850 at A=2 to .757 at A=6 while each arm's share collapses. And sharper routing does not rescue it: tau .10/.05/.02 gave STS-B .7520 / .7514 / .7468 while key separation rose. The value is graded blending, not selection.

So every candidate arm is judged against an untrained arm at the same arm count, not against the smaller base. An empty arm gains +.0045 here -- real generic capacity. Greedy forward selection, bar = the measured seed spread .0054:

Every A=4 candidate -- including the control -- came in below .7684. Selection stopped at three arms. Zero starvation alarms; per-block routing spread .348, so the 12 dispatches learned different decisions rather than 12 copies of one.

Loading arms

Arms attach through the same AutoModel object. amoe-lora is imported lazily, so the base model still loads on a machine that has never heard of it.

from transformers import AutoModel
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2",
                                  trust_remote_code=True)

model.attach_amoe()                       # the shipped 3-arm collective
emb = model.encode(["a cat on a windowsill"])

with model.amoe_off():                    # the unsupervised baseline
    base = model.encode(["a cat on a windowsill"])

model.set_amoe(["equiv"])                 # one arm, inside the dispatch
model.detach_amoe()                       # bit-exact restore, asserted

attach_amoe() with no arguments reads the roster and its order from the dispatch checkpoint -- routing keys are per-arm and positional, so order is not cosmetic. Both attach_amoe and detach_amoe assert the toggle law and raise rather than return a silently-wrong model.

pip install git+https://github.com/AbstractEyes/amoe-lora for the arm methods.

Arms do not transfer between trunks

These anchors were trained against this trunk's residual stream. Measured against the sibling captionbert-8192-b (same architecture, complete 66-chunk corpus):

Transferring costs 31% of the gain; re-training only the 1,536 routing keys recovers 29% of that, so 71% of the loss is in the anchors themselves.

The two trunks are indistinguishable on all eight tasks (+.0009 mean excluding the 100-row BIOSSES) and on geometry, yet 1.6M adapter parameters tell them apart -- adapters read the residual stream, the task gauges read the pooled output, and the stream carries trunk identity the output does not. Budget one anchor set per trunk (~18 min).

Arm library

amoe/arms/ holds one canonical copy of every anchor with ARMS.json (provenance, training spec, solo scores, capacity verdict). Resolve from there; do not retrain what is already listed. The campaign folders (amoe/{sts,sts-combo,moe,moe-v2,collective}) remain as the records behind the published numbers.

Scope worth stating: these anchors are solo-trained and always-on, which the aleph line identifies as blend regime -- the dispatch blends them rather than containing them. That is survivable here only because every arm serves one task. On a multi-task trunk, expect the cross-task interference that record documents. See amoe-lora.

How it was built

1. Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d).
2. One global whitened Procrustes map per teacher into bert-base's frame, fit on a stratified random sample and reported out-of-sample.
3. Consensus = normalized centroid of the aligned teachers, per chunk.
4. Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the consensus. Pure Adam, no weight decay. 26.9M rows, 52,548 steps at batch 2048, ~5.4 h on one RTX 6000 Pro.

Known limits -- read before using

• Consensus rank is ~28.7 of 768. Five BERT-family teachers only agree on about 29 directions. The student uses ~103 in domain but falls back to ~33 on out-of-domain text: the structure it builds on captions does not transfer. This is the model's ceiling and it is a property of the consensus, not the student.
• Alignment quality varies by teacher. Out-of-sample cosine to the bert frame: distil .625, roberta .372, albert .331, modern .327. The ordering tracks architectural distance from bert-base.
• 10 of 66 source chunks lacked ModernBERT, so 54 chunks (~27M rows) were used. No 4-expert fallback: that would change the target definition mid-dataset.
• Trained on image captions. Expect caption-like text to be its strongest domain.
• A sibling trained on the complete 66-chunk corpus (-b) scores within .0046 of this one bare and .0008 with arms. 19% more data bought nothing: the ceiling is teacher agreement (28.7 of 768 shared directions), not corpus size.
• The trunk is single-seed. The AMOE results are 2-seed with a measured spread of .003-.005, against margins of +.012 to +.019.
• amoe/sts-combo stopped at step 500 of a planned 4000; its card and config describe the full run. amoe/moe-v2 retrained its anchors instead of reusing them, so the moe-vs-moe-v2 comparison moved two variables, not one. Both are recorded rather than quietly corrected.

Output convention (differs from v1)

geolip-captionbert-8192 (v1) returned the pooled embedding as last_hidden_state. If porting v1 code, use pooler_output. v1 also shipped an AlignmentBank; v2 does not -- measured on v1, its expert-consistency features varied 0.2% across samples because a rotation round-trip carries no data.

Citation

@misc{abstractphil2026captionbertv2,
  title  = {captionbert-8192-v2: consensus distillation at CC12M scale},
  author = {AbstractPhil},
  year   = {2026},
  url    = {https://huggingface.co/AbstractPhil/captionbert-8192-v2}
}

MIT.