CET-ViT: Causal Emergence Transformer for Vision
Model Description
CET-ViT introduces a learnable causal emergence operator (V-CEO) inside a hierarchical Vision Transformer. Instead of fixed pooling, V-CEO learns a soft assignment matrix S that groups patch tokens into macro visual regions guided by Effective Information (EI) — a measure of causal power.
Key innovation: The first end-to-end trainable causal emergence operator inside a ViT, using SVD-based dynamical reversibility as a differentiable EI proxy (Yuan et al., npj Complexity 2025).
Results on CIFAR-100 (training from scratch)
| Model | Params | Val OA | Val F1-macro | Reversibility |
|---|---|---|---|---|
| CET-ViT (ours, with EI) | 33M | 73.32% | 73.12% | 0.9162 |
| CET-ViT ablation (no EI) | 33M | TBD | TBD | ~0.35-0.45 |
Causal Emergence Probing
- Mean reversibility: 0.9162 ± 0.012 (STRONG causal emergence)
- Mean K selected: 8.4 / 32 (model self-organized scale)
- EI trajectory: -0.48 (init) → -0.87 (epoch 300)
Architecture
- Scale: base (embed_dim=192, depths=[2,6,6,2])
- Parameters: 33M
- V-CEO: Dynamic K selection via Gumbel-softmax + soft assignment S
- Loss: CrossEntropy + λ·SVD-EI loss (cosine warmup epochs 50-200)
- Training: 300 epochs, batch 128, Mixup α=0.8, AutoAugment, RandomErasing
Training Config
make_model_config(
scale="base", img_size=32, patch_size=2, num_classes=100,
drop_path_rate=0.3, drop_rate=0.1, attn_drop_rate=0.1,
k_candidates=[4, 8, 16, 32],
)
# lambda_ei_end=0.05, warmup_epochs=50-200 cosine
# lr=5e-4, weight_decay=0.1, AdamW
Citation
@article{cetvit2026,
title={CET-ViT: Causal Emergence Transformer for Vision},
author={Dhruv Jyoti Das},
year={2026}
}