TopoVoid-0.2B-Orthogonal-Preview

• Organization: Topovoid (topovoid.com)
• Maintainers: tirpitzia, 0kaguya
• License: Apache-2.0

1. Overview

This repository contains the 0.2B research preview (v0.1) of the TopoVoid architecture. It is a 212M-parameter bilingual causal language model pretrained from scratch with explicit middle-layer concept bottlenecks, paired with a closed-form solver that writes new facts into Attention and MLP weights inside the orthogonal null space of those concept layers.

How the closed-form write works (closed_form_write.py):

1. Trigger contexts: Each fact (subject, declarative prompt, target, optional question) is expanded into declarative and QA prompt variants. Multi-token targets are unrolled into sequential token continuation steps.
2. Pullback target ($R$): For each edited layer ($\ell \in {8, 9, 10, 11}$, 1-based), the model runs forward to layer $\ell$ and takes three normalized autograd steps on the hidden activation to find a residual $R$ that increases the target token probability. Autograd is used to compute this activation target before the linear solve; no optimizer step or gradient descent updates the model weights.
3. Null-space projection ($P_\perp$): The residual is projected through $$P_\perp = I_{1024} - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T, \qquad P_\perp Q_{\mathrm{cbm}} = 0$$ where $Q_{\mathrm{cbm}} \in \mathbb{R}^{1024 \times 176}$ is the orthonormal basis spanning the read and write matrices of the L7 MidCBM, L7 SubspaceTopKCBM, and L12 Highway CBM. Of the 1024 residual dimensions, 176 (17.2%) are protected and 848 (82.8%) form the writable null space ($\max |P_\perp Q_{\mathrm{cbm}}| = 1.47 \cdot 10^{-7}$ in fp32).
4. Closed-form linear solve ($V$): Trigger keys, natural-text covariance (eval_c4, eval_zh in vocab_map_49152_chatml.npz), and negative/continuation suppression keys are combined in a regularized linear system to solve for $V$, yielding low-rank weight increments $\Delta W = P_\perp R V$ on the Attention output projection and MLP down-projection.
5. Mounting and unmounting: During multi-layer compilation, earlier layers' increments are applied before solving later layers. At inference, a compiled module is added directly to the base weights (W + ΔW), adding zero prompt tokens.

Architecture

Files in This Repository

---

2. Quick Start & Open-Dataset Benchmark

Tested with torch 2.8.0, transformers 4.57.1, safetensors 0.8.0, and numpy on one GPU.

from modeling_topovoid import load_topovoid
from closed_form_write import concept_null_projector

model, tok, o2c, c2o = load_topovoid('.')
P_perp, Q_cbm = concept_null_projector(model)

Run the minimal verification and closed-form write demo (demo_write.py), or pass --counterfact to evaluate batch compilation on counterfact.json:

python demo_write.py
python demo_write.py --counterfact counterfact.json --n 12

Summary of Open-Dataset Results

The three evaluation tracks below test distinct mechanisms—direct weight writing, frozen-feature linear readout, and external concept-index retrieval—and should not be conflated:

Example raw output on a CounterFact edit (target Toronto, greedy decoding, max 16 new tokens):

Prompt       : Sunrise Party is headquartered in
Before write :  the heart of the city. The club is a 100% free
After write  :  Toronto, Canada. The company is based in Toronto, Canada, and is a

---

2.1 CounterFact (1-Hop Weight Write)

First edit layer: Layer 8 (1-based). CounterFact metrics below use first-target-token proxies of ES / PS / NS (official ROME scores full target sequences):

• ES: $P(\mathrm{new}) > P(\mathrm{true})$ on the rewrite prompt.
• PS: $P(\mathrm{new}) > P(\mathrm{true})$ on paraphrase prompts.
• NS: $P(\mathrm{true}) > P(\mathrm{new})$ on neighborhood prompts (higher means less bleed into related subjects).
• Top-1: First-token top-1 hit rate of the new target on rewrite, paraphrase, and neighborhood prompts.

2.2 Stacking Modules

Each module contains 40 CounterFact facts compiled independently from the same base checkpoint and added linearly (W + Σ ΔW_i):

Note on multi-module decay and the 1B compiler upgrade: In this 0.2B preview repository, closed_form_write.py only projects each module individually against the protected concept basis $Q_{\mathrm{cbm}}$ ($P_\perp = I - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T$) and does not include a cross-module Gram-Schmidt joint orthogonalization operator across different modules' right-hand key subspaces. As a result, 2 stacked modules (80 facts) retain 96.25% Top-1 (0.997886 Highway cosine), whereas summing 4 or 8 independently solved modules causes key collisions inside the null space (55.62% and 34.69%). Cross-module Gram-Schmidt joint compilation is implemented in our internal 1B commercial compiler (see Section 3).

2.3 MQuAKE-CF-3k (2-Hop)

Evaluated on 150 sampled cases (149 allow the unrelated-subject control). These numbers do not establish multi-hop reasoning on the base checkpoint:

2.4 Text Classification (AG News & DBpedia-14)

A separate ridge classifier head is fitted on frozen representations without changing the language model weights or generation path. Full L7 (1024-d) and CBM (320-d, MidCBM + SubspaceTopKCBM) are reported separately; Full L7 results cannot be attributed to the concept bottleneck alone.

Fixed-Hyperparameter 40-Label Test (AG News, 10 Labels/Class, 0 Validation Labels)

With ridge $\lambda = 0.1$ and LinearSVC $C = 1.0$ fixed in advance across 5 sampling seeds (20261203–20261207) and no validation labels used for selection:

• Full L7 (1024-d): 75.37%
• CBM (320-d): 67.32%
• Word (1–2 gram) + Character char_wb (3–5 gram) TF-IDF LinearSVC: 56.70%
• Difference (Full L7 − SVM): +18.67 pp (95% paired seed-$t$ CI [+15.75, +21.60] pp; Holm-corrected $p = 0.0002975$).

Five-Seed Low-Label Curve (With 8,000-Example Validation Selection)

Each method uses the exact same training subsets (seeds 20261203–20261207) and selects hyperparameters on the 8,000-item validation split:

Full-Dataset Readout & Perturbation Checks

AG News uses 112,000 fit / 8,000 val / 7,600 test items. DBpedia-14 uses 120,000 fit / 8,000 val / 70,000 test items (10,000-item test subset for zero-shot rows; on that same 10,000-item subset, CBM / Full L7 / TF-IDF score 90.96% / 96.53% / 98.23%):

Under surface perturbations on AG News (evaluated without retraining the head): Original 89.00% (TF-IDF 91.82%), All lowercase 86.45% (91.82%), Strip Reuters/AP datelines 88.83% (91.45%), First 32 tokens 86.09% (90.75%). The 4-class L7 head is 40 KiB (32 KiB float64 weights + 8 KiB float32 mean/std). Full-split tokenization takes 15.89 s, feature extraction 373.46 s, and ridge fitting 1.75 s per $\lambda$ candidate.

2.5 External MidCBM Retrieval Index + Copy Head

On 399 unseen paraphrases of stored ARC science questions, indexing frozen MidCBM and SubspaceTopKCBM states in an external retrieval index with a zero-parameter copy head raises accuracy from 28.57% (0.2B alone) to 72.93% (+44.36 pp, 95% CI [+39.36, +49.38] pp; 72.68% with shuffled options; 96.3% precision when triggered during 24-token greedy generation; 1.000009 max PPL ratio on C4/ZH).

---

3. Further Research & Next-Gen Multimodal Results

Based on this early 0.2B orthogonal framework, our team has completed further internal work on cross-modal representation alignment and industrial compilation toolchains (including cross-module Gram-Schmidt joint orthogonalization), scaling this formulation to 1B / 3B / 7B multimodal architectures.

On our 1B model (currently in testing and rollout across production workflows), mounting closed-form orthogonal modules without fine-tuning base weights yields:

• Enterprise Email Archiving & Routing: 65.8% → 98.5%
• Complex Scanned Document Classification: 64.2% → 97.6%
• SAR / Satellite Imagery Classification (used by OSU researchers): 64.7% → 98.3%

Technical details and industrial pipelines (CAD parsing and Tech Pack breakdown) are available at https://topovoid.com.

---

4. Limitations

• Readout heads are separate modules. A classification head reads L7 features and outputs class scores directly. It does not change what the language model generates and does not show general reasoning gains.
• Cosine 1.0 on L7 MidCBM comes from causal order as well as projection. With the first edit at Layer 8, L7 MidCBM and SubspaceTopKCBM activations are computed upstream of the edited weights and cannot change. The L12 Highway CBM is downstream of layers 8–11 and is not strictly invariant (0.9979 → 0.9890 mean cosine across 2 to 8 stacked modules).
• Module stacking interferes beyond 2 modules in v0.1. In this 0.2B preview, closed_form_write.py only performs single-module orthogonal projection against $Q_{\mathrm{cbm}}$ ($P_\perp = I - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T$) without a cross-module Gram-Schmidt joint orthogonalization operator. While 2 stacked modules (80 facts) retain 96.25% Top-1, adding 4 or 8 independently compiled modules degrades Top-1 to 55.62% and 34.69% and raises perplexity. Cross-module joint orthogonal compilation is part of our internal 1B commercial compiler.
• Report paraphrase and neighborhood metrics beside rewrite Top-1. Rewrite Top-1 measures reproduction of the written prompt. Paraphrase generalization is lower (35.0%–41.7% Top-1), and edits bleed into neighboring subjects of the same relation (27.5%–56.7%).
• Multi-hop hits require the control. On MQuAKE-CF-3k, 117/150 multi-hop questions hit the edited answer, but 118/149 unrelated-subject control questions also trigger the same answer via surface question templates.
• Writes raise perplexity on unrelated text, scaling with the number of facts written into a single module.
• The external memory index only recalls stored items, and can copy a wrong stored answer when retrieval is confident but incorrect.
• Pruned vocabulary. Unsupported tokenizer IDs map to compact ID 0 in ordinary forward passes; closed_form_write.py splits targets into active subtokens where possible.
• Base model without safety tuning. Pretraining stages used web text without spam or adult-content filtering, and no chat or safety alignment was applied. Do not deploy directly in user-facing production.

---

5. License & Attributions

License

Code and TopoVoid-0.2B-Orthogonal-Preview weights in this repository are released under the Apache License 2.0 (see LICENSE).

• Organization: https://huggingface.co/Topovoid
• Developers: https://huggingface.co/tirpitzia, https://huggingface.co/0kaguya

Attributions

• Tokenizer and distillation teachers: Qwen tokenizer and Qwen teacher models (Qwen, Apache-2.0)
• Pretraining and calibration text: FineWeb, FineWeb2 (ODC-By 1.0), and C4 (ODC-By 1.0) (eval_c4, eval_zh in vocab_map_49152_chatml.npz)
• Evaluation benchmarks: CounterFact (MIT), MQuAKE (MIT), AG News, DBpedia-14 (CC-BY-SA-3.0), and AI2 ARC (CC-BY-SA-4.0)

Citation

@misc{topovoid2026orthogonal,
  title        = {TopoVoid-0.2B-Orthogonal-Preview: Native Orthogonal Subspace Architecture and Closed-Form Null-Space Weight Compilation},
  author       = {tirpitzia and 0kaguya and {TopoVoid Research}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview}},
  note         = {Research Preview v0.1, \url{https://topovoid.com}}
}