RoBERTa Base with AdaSplash Attention via NeuralOps

Model Details

• Model Identifier: Hooshaai/roberta-base-adasplash
• Base Architecture: RoBERTa Base fine-tuned on GLUE SST-2 (125M parameters, 12 layers, 12 heads, hidden size 768)
• Attention Mechanism: adasplash_attention (Adaptive Splash Attention with Tsallis $\alpha=1.5$ Entmax Sparse Simplex Projection)
• Attention Sparsity: 40% exact mathematical zero weights ($A_{ij} \equiv 0$) across all 12 encoder blocks
• Parameters: 125.8M (+0.94% parameter count via AdaSplashLinear residual LoRA adapters)
• Framework: neuralops v1.4.3
• Organization: Hooshaai

Architecture Overview

RoBERTa Base relies on deep bidirectional contextual representations across 12 transformer encoder blocks. Because RoBERTa features byte-pair encoding with larger sequence vocabularies, dense Softmax attention tends to over-allocate probability mass to punctuation, subword tokens, and padding symbols.

AdaSplash Attention incorporates Tsallis 1.5-entmax into RoBERTa's multi-head attention blocks:

1. 40% Exact Attention Sparsity: Replaces dense softmax with Tsallis $\alpha=1.5$ entmax, projecting attention distributions onto the simplex boundary and ensuring that 40% of attention matrix entries are strictly zero.
2. LoRA-Augmented Residual Projections: Projections ($W_Q, W_K, W_V, W_O$) are implemented via AdaSplashLinear, freezing pre-trained weights while learning additive rank-16 adapter pathways.
3. Full Encoder Context Retention: Rather than applying local causal splash windowing, the bidirectional encoder retains full receptive field reach while filtering irrelevant cross-attention via mathematical thresholding.

Input Activation X [B, L, 768]
       │
       ├── AdaSplashLinear Projections: Q, K, V ∈ ℝ^{B × 12 × L × 64}
       ├── Normalized Scaled Dot-Products: S_{i, j} = (Q_i K_j^T) / 8
       ├── Exact Tsallis 1.5-Entmax Simplex Projection:
       │     Solves: ∑ max(0, (S_{i, j} - τ)/2)^2 = 1 via bisection search
       │     Produces A_{i, j} with 40% exact zeros
       ├── Sparse Context Aggregation: O_i = ∑_{j: A_{i, j} > 0} A_{i, j} V_j
       └── Output Synthesis: Y = AdaSplashLinear_O(O)  [B, L, 768]

Mathematical Formulation & Derivation

1. Entmax Projection on RoBERTa Hidden States

Given logit vector $s \in \mathbb{R}^L$ for each query position: [ p^(s) = \arg\max_{p \in \Delta^{L-1}} \left{ p^\top s + \frac{4}{3} \sum_{i=1}^L (p_i - p_i^{1.5}) \right} ] The analytical solution is: [ p_i^(s) = \left[ \frac{s_i - \tau}{2} \right]_+^2 ] where $\tau$ is uniquely determined by monotonic root-finding on the simplex constraint.

2. Gradient Flow and Non-Vanishing Derivatives

Unlike hard step thresholding, the gradient of 1.5-entmax with respect to logits is continuous and Lipschitz bounded: [ \frac{\partial p_i^}{\partial s_j} = \left[ \frac{s_i - \tau}{2} \right]+ \left( \delta{ij} - \frac{\left[ \frac{s_j - \tau}{2} \right]+}{\sum{k \in \operatorname{supp}(p^)} \left[ \frac{s_k - \tau}{2} \right]_+} \right) ] guaranteeing stable gradient backpropagation into the pre-trained encoder weights and LoRA adapters.

Empirical Benchmark Table

Evaluated on the GLUE SST-2 validation split on NVIDIA GeForce RTX 4070 (SM 8.9) with sequence length $L = 128$ and batch size $B = 32$:

Quickstart Code

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from neuralops import replace_attention

# 1. Load tokenizer and model from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-adasplash")

# 2. Or surgically inject AdaSplash into any RoBERTa in 1 line
base_model = AutoModelForSequenceClassification.from_pretrained("roberta-base")
adasplash_model = replace_attention(base_model, "adasplash_attention")

# 3. Predict sentiment
inputs = tokenizer("The sparse entmax projection preserves salient features with extreme precision!", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    prediction = logits.argmax(-1).item()

print(f"Predicted class: {prediction} ({'Positive' if prediction == 1 else 'Negative'})")

Hardware & Software Requirements

• PyTorch: 2.11+
• CUDA: 12.8+
• Triton: 3.6+
• Accelerators: NVIDIA SM 8.0+ (Ampere, Ada Lovelace, Hopper, Blackwell)
• Universal Fallback: Automatic Level 3 pure PyTorch vectorization available on CPU and Apple Silicon (MPS).

Citation

@software{neuralops2026,
  author = {Majlesi, Taha and contributors},
  title = {NeuralOps: Unified Platform for Subquadratic Attention and Deep Model Compression},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/tahamajs/NeuralOps}},
  version = {1.4.3}
}