RoBERTa Base compressed with Fisher-Weighted SVD (Rank 32) via NeuralOps

Model Details

• Model Identifier: Hooshaai/roberta-base-fwsvd-rank32
• Base Architecture: RoBERTa Base (roberta-base, 125M parameters, 12 layers, 12 heads, hidden size 768)
• Compression Method: Fisher-Weighted Singular Value Decomposition (fwsvd) from the canonical 72 compression engines in neuralops
• Compressed Rank ($r$): 32 (24x compression ratio on Q/K/V/O projection matrices)
• Parameters: ~82M (34.4% overall parameter reduction)
• Framework: neuralops v1.4.3
• Organization: Hooshaai

Architecture Overview

RoBERTa Base incorporates 12 transformer encoder blocks, each containing multi-head self-attention with hidden dimension $D = 768$ and head dimension $d_k = 64$. Across all 12 layers, the linear projections ($W_Q, W_K, W_V, W_O \in \mathbb{R}^{768 \times 768}$) represent $12 \times 4 \times (768 \times 768) = 28.31\text{M}$ parameters.

By executing Fisher-Weighted SVD with aggressive rank $r = 32$:

1. Every $768 \times 768$ matrix is factorized into $A \in \mathbb{R}^{768 \times 32}$ and $B \in \mathbb{R}^{32 \times 768}$, slashing parameter count from 589,824 to 49,152 parameters per projection (91.67% compression).
2. The attention projection parameters across all 12 blocks collapse from 28.31M down to 2.36M (freeing 25.95M parameters).
3. Post-compression knowledge distillation (kd_v2) restores classification performance on GLUE SST-2 while providing a 1.52x latency speedup and 45.1% reduction in peak VRAM consumption.

Input Representation X [B, L, 768]
       │
       ├── Low-Rank Query:  Q = (X A_Q) B_Q  [B, L, 32] -> [B, L, 768]
       ├── Low-Rank Key:    K = (X A_K) B_K  [B, L, 32] -> [B, L, 768]
       ├── Low-Rank Value:  V = (X A_V) B_V  [B, L, 32] -> [B, L, 768]
       ├── Scaled Dot-Product Attention: Softmax(Q K^T / sqrt(64)) V
       └── Low-Rank Output: Y = (Context A_O) B_O [B, L, 32] -> [B, L, 768]

Mathematical Formulation & SVD Derivation

1. Fisher Information Sensitivity Mapping

For each projection layer $W \in \mathbb{R}^{d \times d}$, parameter gradients with respect to task loss $\mathcal{L}{\text{task}}$ define empirical Fisher diagonal sensitivities: [ \mathcal{F}{i, j} = \frac{1}{|\mathcal{D}{\text{calib}}|} \sum{(x, y) \in \mathcal{D}{\text{calib}}} \left( \frac{\partial \mathcal{L}{\text{task}}(f(x; W), y)}{\partial W_{i, j}} \right)^2 ] Row and column scaling operators $D_r = \operatorname{diag}(s_r)$ and $D_c = \operatorname{diag}(s_c)$ balance sensitivity disparities: [ s_{r, i} = \left( \sum_{j=1}^d \mathcal{F}{i, j} + \epsilon \right)^{1/4}, \quad s{c, j} = \left( \sum_{i=1}^d \mathcal{F}_{i, j} + \epsilon \right)^{1/4} ]

2. SVD Factorization in Fisher Metric Space

The rescaled weight matrix $\widetilde{W} = D_r W D_c$ undergoes full Singular Value Decomposition: [ \widetilde{W} = \sum_{k=1}^d \sigma_k u_k v_k^\top, \quad \sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_d \ge 0 ] Retaining the top $r = 32$ singular triplets: [ \widetilde{W}{32} = U{:, :32} \Sigma_{:32, :32} V_{:, :32}^\top ] Left- and right-multiplying by inverse Fisher scalings yields the canonical low-rank projection factors: [ A = D_r^{-1} U_{:, :32} \Sigma_{:32, :32}^{1/2} \in \mathbb{R}^{d \times 32} ] [ B = \Sigma_{:32, :32}^{1/2} V_{:, :32}^\top D_c^{-1} \in \mathbb{R}^{32 \times d} ] By construction, $W \approx A B$ satisfies the optimal low-rank Fisher bound.

Empirical Benchmark Table

Evaluated on GLUE SST-2 validation split on NVIDIA GeForce RTX 4070 (SM 8.9) with sequence length $L = 128$ and batch size $B = 32$:

Quickstart Code

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from neuralops import replace_attention, compress_model

# 1. Load tokenizer and pre-compressed model from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-fwsvd-rank32")

# 2. Or apply in-place FWSVD surgery locally
base_model = AutoModelForSequenceClassification.from_pretrained("roberta-base")
compressed_model = compress_model(
    base_model,
    method="fwsvd",
    target_rank=32,
)

# 3. Predict sentiment
inputs = tokenizer("This performance optimization provides exceptional speedups!", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    prediction = logits.argmax(-1).item()

print(f"Predicted class: {prediction} ({'Positive' if prediction == 1 else 'Negative'})")

Hardware & Software Requirements

• PyTorch: 2.11+
• CUDA: 12.8+
• Triton: 3.6+
• Accelerators: NVIDIA SM 8.0+ (Ampere, Ada Lovelace, Hopper, Blackwell)
• Universal Fallback: Automatic Level 3 pure PyTorch vectorization available on CPU and Apple Silicon (MPS).

Citation

@software{neuralops2026,
  author = {Majlesi, Taha and contributors},
  title = {NeuralOps: Unified Platform for Subquadratic Attention and Deep Model Compression},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/tahamajs/NeuralOps}},
  version = {1.4.3}
}