加载中…
加载中…
BERT-base (110M params) trained in 6 variants (3 BWSK modes x 2 experiments) on WikiText-2 with full convergence training and early stopping.
This repo contains all model weights, configs, and training results in a single consolidated repository.
BWSK is a framework that classifies every neural network operation as S-type (information-preserving, reversible, coordination-free) or K-type (information-erasing, synchronization point) using combinator logic. This classification enables reversible backpropagation through S-phases to save memory, and CALM-based parallelism analysis.
| Property | Value |
|---|---|
| Base Model | google-bert/bert-base-uncased |
| Architecture | Transformer (masked_lm) |
| Parameters | 110M |
| Dataset | WikiText-2 |
| Eval Metric | Pseudo-Perplexity |
| Type | Ratio |
|---|---|
| S-type (information-preserving) | 67.3% |
| K-type (information-erasing) | 32.7% |
| Mode | Final Loss | Val Pseudo-Perplexity | Test Pseudo-Perplexity | Peak Memory | Time | Epochs |
|---|---|---|---|---|---|---|
| Conventional | 1.8896 | 5.56 | 5.40 | 4.0 GB | 7.4m | 5 |
| BWSK Analyzed | 1.9163 | 5.54 | 5.57 | 4.0 GB | 7.3m | 5 |
| BWSK Reversible | 1.5086 | 5.57 | 5.49 | 2.9 GB | 9.1m | 5 |
Memory savings (reversible vs conventional): 27.7%
| Mode | Final Loss | Val Pseudo-Perplexity | Test Pseudo-Perplexity | Peak Memory | Time | Epochs |
|---|---|---|---|---|---|---|
| Conventional | 6.9915 | 1383.85 | 1489.18 | 4.0 GB | 7.3m | 5 |
| BWSK Analyzed | 7.4792 | 1373.72 | 1480.62 | 4.0 GB | 7.4m | 5 |
| BWSK Reversible | 7.0919 | 1401.24 | 1503.86 | 2.9 GB | 9.0m | 5 |
Memory savings (reversible vs conventional): 27.6%
├── README.md
├── results.json
├── finetune-conventional/
│ ├── model.safetensors
│ ├── config.json
│ └── training_results.json
├── finetune-bwsk-analyzed/
│ ├── model.safetensors
│ ├── config.json
│ └── training_results.json
├── finetune-bwsk-reversible/
│ ├── model.safetensors
│ ├── config.json
│ └── training_results.json
├── scratch-conventional/
│ ├── model.safetensors
│ ├── config.json
│ └── training_results.json
├── scratch-bwsk-analyzed/
│ ├── model.safetensors
│ ├── config.json
│ └── training_results.json
├── scratch-bwsk-reversible/
│ ├── model.safetensors
│ ├── config.json
│ └── training_results.json
Load a specific variant:
from transformers import AutoModelForMaskedLM, AutoTokenizer
# Load fine-tuned conventional variant
model = AutoModelForMaskedLM.from_pretrained(
"tzervas/bwsk-bert-base", subfolder="finetune-conventional"
)
tokenizer = AutoTokenizer.from_pretrained(
"tzervas/bwsk-bert-base", subfolder="finetune-conventional"
)
# Load from-scratch BWSK reversible variant
model = AutoModelForMaskedLM.from_pretrained(
"tzervas/bwsk-bert-base", subfolder="scratch-bwsk-reversible"
)
| Setting | Value |
|---|---|
| Optimizer | AdamW |
| LR (fine-tune) | 5e-05 |
| LR (from-scratch) | 3e-04 |
| LR Schedule | Cosine with warmup |
| Max Grad Norm | 1.0 |
| Mixed Precision | AMP (float16) |
| Early Stopping | Patience 3 |
| Batch Size | 4 |
| Sequence Length | 512 |
@software{zervas2026bwsk,
author = {Zervas, Tyler},
title = {BWSK: Combinator-Typed Neural Network Analysis},
year = {2026},
url = {https://github.com/tzervas/ai-s-combinator},
}
MIT