GEOLIP-CLIP-ViT-L/14 ctx576 — Memory-Extended CLIP Text Encoder

Extends CLIP-ViT-L/14 text encoder from 77 tokens to 576 effective context via geometric memory bank, trained by distillation from ModernBERT-large.

Key Results

Usage

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "AbstractPhil/geolip-clip-vit-large-patch14-ctx576",
    trust_remote_code=True)
model = model.to("cuda").eval()

# Single text → 768-dim CLIP-compatible embedding
embedding = model.encode("A vast sweeping landscape of rolling green hills...")
print(embedding.shape)  # (768,)

# Batch
embeddings = model.encode([
    "A cat sleeping on a warm blanket",
    "A medieval castle overlooking a turbulent sea with waves crashing "
    "against ancient rocks and seabirds wheeling overhead against a sky "
    "painted in shades of grey and gold as the sun breaks through clouds"
])
print(embeddings.shape)  # (2, 768)

# HuggingFace forward() API
output = model(texts=["A photo of a dog"])
print(output.last_hidden_state.shape)  # (1, 1, 768)

Architecture

Long caption (576 tokens)
    │
    ├── ModernBERT-large ──── 4096 ctx, 1024-dim ──── teacher (training only)
    │   (frozen, 395M)
    │
    └── CLIP-L text enc ──── 18 tok × 32 segments ── student
        (frozen, 768-dim, 123M)
        + Geometric Memory System (34.5M trainable)
            ├── Depth compressor (6-layer profile → 768-dim anchor)
            ├── Memory bank (64 anchors, 2-layer cross-attention)
            ├── CLIP cross-attention (memory → CLIP hidden states)
            ├── GRU gate (memory state updates)
            ├── Layer fusion (learned weighted sum of 6 CLIP layers)
            └── Teacher projector (768 → 1024, Procrustes-initialized)

How It Works

1. Long text split into 18-token segments (stride 14, overlap 4)
2. Each segment: frozen CLIP text encoder forward (standard, untouched)
3. Memory tokens cross-attend to CLIP's multi-layer hidden states
4. Depth-profile anchors stored in geometric memory bank
5. GRU gate controls rolling memory state across segments
6. Final output: 768-dim embedding in CLIP's text space

Training

• Teacher: ModernBERT-large (frozen, 4096 context)
• Losses: InfoNCE(student→teacher) × 1.0 + SVD Procrustes regularizer × 0.3 + pentachoron CV × 0.05
• Data: CC12M with LLaVA-next detailed captions (50K train, 2K val, mean 96 tokens)
• Hardware: NVIDIA RTX PRO 6000 Blackwell (102 GB VRAM)
• Training time: ~103 minutes (10 epochs × ~10.3 min/epoch)
• Batch size: 64

Geometric Properties

• Procrustes init: CLIP→ModernBERT cos 0.001 → 0.816 (strongest alignment seen across all GEOLIP experiments)
• CV trajectory: 0.185 (E1) → 0.162 (E10), stable in the 0.16-0.19 band
• CV regularization on bank anchors prevented projector shortcut collapse
• 18-token segments forced genuine memory accumulation (vs 55-token segments which plateaued at 0.670)

Why 18-Token Segments?

55-token segments → 2 segments per caption → pentachoron CV dead (needs 5+ anchors) → projector shortcut → plateau at m_acc=0.670.

18-token segments → 7 segments per caption → CV active → memory bank forced to accumulate → m_acc=0.945.

The geometry regularization must be applied at the bottleneck (bank anchors). If it can't activate, the model finds cheap projector shortcuts.

Training Metrics

Full training metrics available in metrics.json. TensorBoard logs in tensorboard/.

More than likely the model didn't converge yet and requires additional data. Lots more data. It's a good start though.

Training Curve

*E2 val dip: cosine LR transient at epoch boundary, recovered by E3.

GEOLIP Family

All use the same blueprint: frozen expert teacher + Procrustes initialization + InfoNCE alignment force + pentachoron CV on the bottleneck representation.

License

Apache 2.0