SALTAttention: SALT Attention module replacing standard linear projections with SALTLinear. Hardened for production across DistilBERT, RoBERTa, GPT-2, and Qwen3.5. Supports Fisher-weighted SVD, attention-aware gamma, and adaptive rank (SALT-F).
This repository packages a modular NeuralOps LLaVA-Qwen-0.5B Multimodal Attention Surgery Zoo checkpoint derived from llava-hf/llava-interleave-qwen-0.5b-hf using salt_attention and evaluated on ScienceQA (Image-Bearing STEM Subset, validation).
| Property | Specification |
|---|---|
| Zoo Track | llava_vlm (LLaVA-Qwen-0.5B Multimodal Attention Surgery Zoo) |
| Operator / Registry Key | salt_attention (SALTAttention, family svd) |
| Source Implementation | neuralops/layers/svd/salt_attention.py |
| Base Architecture | llava-hf/llava-interleave-qwen-0.5b-hf — Frozen SigLIP Vision Tower + 2-layer MLP Projector + 24-layer Qwen2 decoder (D=1024, H=16, head_dim=64) |
| Surgical / Compression Scope | 18 / 24 decoder/encoder layers (hybrid_3_1) |
| Modified & Saved Components | 18/24 decoder language_model.model.layers.{i}.self_attn + trained decoder layernorms |
| Frozen & Excluded Components | Frozen SigLIP vision_tower.* (27 layers), multi_modal_projector.*, 6 quadratic anchor layers [3, 7, 11, 15, 19, 23], decoder MLPs & embeddings |
| Metric | Teacher Baseline | Pre-KD / Zero-Shot | Post-KD / Final | Delta ($\Delta$) |
|---|---|---|---|---|
| Accuracy (%) | 63.75% |
31.25% |
33.12% |
+1.87% |
| Perplexity (PPL) | 2.45 |
14.07 |
3.04 |
-11.03 |
295 (Stage 1: 133 | Stage 2: 162)120.4s16489 MB2026-11-11T09:35:10.915863Zattention_weights.safetensors46.40 MB (48,655,776 bytes) | Total Package Size: 61.59 MBllava-hf/llava-interleave-qwen-0.5b-hf, 1.66 GB): 1.62 GB saved (-97.3%, 36.67x smaller artifact)surgical_manifest.json): schema_version=neuralops.surgery/v2, saved_tensor_count=606, weights_format=attention_only_fp16| File on Disk | Size | Bytes | Role |
|---|---|---|---|
attention_weights.safetensors |
46.40 MB |
48,655,776 |
Modular surgical / compressed delta weights (FP16 safetensors) |
surgical_manifest.json |
51.7 KB |
52,929 |
Surgical / compression topology and modified-prefix manifest |
config.json |
6.4 KB |
6,573 |
Model architecture and NeuralOps surgery/compression metadata |
training_metrics.json |
767 B |
767 |
Machine-readable evaluation, telemetry, and distillation metrics |
added_tokens.json |
101 B |
101 |
Additional vocabulary tokens |
chat_template.json |
700 B |
700 |
Chat formatting template |
merges.txt |
1.59 MB |
1,671,853 |
BPE merge table |
preprocessor_config.json |
393 B |
393 |
Vision-language image preprocessor configuration |
processor_config.json |
134 B |
134 |
Checkpoint / tokenizer metadata |
special_tokens_map.json |
367 B |
367 |
Special token mappings |
tokenizer.json |
10.89 MB |
11,418,450 |
Fast tokenizer vocabulary and merges |
tokenizer_config.json |
1.6 KB |
1,602 |
Tokenizer special tokens and configuration |
vocab.json |
2.65 MB |
2,776,833 |
BPE vocabulary mapping |
import torch
from PIL import Image
from neuralops.hub import load_surgeried_model
model, processor = load_surgeried_model("tahamajs/llava-qwen-0.5b-salt-attention-scienceqa")
model.eval()
image = Image.new("RGB", (384, 384), color=(255, 255, 255))
inputs = processor(
text="<image>\nQuestion: What is shown in the figure?\nAnswer with the option letter:",
images=image,
return_tensors="pt",
)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=16)
print(processor.decode(outputs[0], skip_special_tokens=True))
Licensed under the MIT License (LICENSE).
@software{neuralops2026vlmzoo,
title = {NeuralOps: Multimodal Vision-Language Subquadratic Attention Surgery Zoo (LLaVA-ScienceQA)},
author = {Majlesi, Taha},
year = {2026},
url = {https://github.com/tahamajs/NeuralOps},
note = {Hub Repository: tahamajs/llava-qwen-0.5b-salt-attention-scienceqa}
}
llava-hf/llava-interleave-qwen-0.5b-hf