⚡ SVD Linear Attention: DistilBERT × A3

This repository contains factorized and validated model weights for DistilBERT with a3 post-training compression and attention factorization, benchmarked on GLUE.

---

📊 Benchmark & Performance Metrics

---

🛠️ Architecture Details

• Backbone Architecture: DistilBERT (6 Transformer Layers, 12 Heads, 768 Hidden Dim)
• Module Engine: a3
• Factorization Stage: Linear Attention Projections ($Q, K, V, O$) Decomposed via Singular Value Decomposition
• Numerical Conditioning: Symmetric $\sqrt{\Sigma}$ Singular Value Allocation to ensure FP16 numerical stability without loss of representation rank.

---

💻 Quickstart & Reproducibility

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo_id = "Hooshaai/svd-linear-attention-roberta-a3"

# 1. Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

# 2. Download Factorized Checkpoint
weights_path = hf_hub_download(repo_id=repo_id, filename="weights.pt")
state_dict = torch.load(weights_path, map_location="cpu")
print(f"Successfully loaded {len(state_dict)} factorized layer tensors from {repo_id}.")

---

📚 Citation & Framework Reference

@article{svd_linear_attention_2026,
  title   = {Unified SVD Linear Attention and Post-Training Factorization in Large Language Models},
  author  = {Hoosha AI Research Team},
  journal = {arXiv preprint},
  year    = {2026}
}