This repository provides a multimodal Vision-Language Model (VLM) derived from llava-hf/llava-interleave-qwen-0.5b-hf (SigLIP vision encoder + MLP projector + 24-layer Qwen2 decoder, D=1024, H=16) where 18 / 24 decoder attention blocks (hybrid_3_1) have been replaced with asvd_attention (ASVDAttention) and distilled on image-bearing questions from derek-thomas/ScienceQA.
ASVDAttention: Hardened ASVD Attention with Factored Low-Rank Matrix Decomposition, GQA, RoPE, and LoRA. Computational Complexity: - Factorized Projections: $\mathcal{O}(B \cdot N \cdot r \cdot (d + d_{\text{all}}))$ - Attention Scoring: $\mathcal{O}(B \cdot H \cdot N^2 \cdot d_k)$ Tensor Dimensions: - hidden_states: $[B, N, D]$ where $B$ is batch size, $N$ is sequence length, and $D$ is hidden dimension. - Output: Tuple of $([B, N, D], \text{None})$.
| Property | Specification |
|---|---|
Registry Key (ATTENTION_REGISTRY) |
asvd_attention |
| NeuralOps Class & Family | ASVDAttention (svd) |
| Source Implementation | neuralops/layers/svd/asvd_attention.py |
| Vision Tower (Frozen) | SigLIP Vision Encoder + 2-layer Multimodal MLP Projector |
| Language Decoder Geometry | 24 layers, D=1024, H=16, d_h=64 (exact power-of-2 widths) |
In LlavaForConditionalGeneration, visual patch tokens extracted by the frozen SigLIP vision tower are interleaved with text tokens inside language_model.model.layers. Keeping vision_tower and multi_modal_projector completely frozen isolates how well subquadratic mechanism asvd_attention handles dense visual-to-text cross-modal routing under hybrid_3_1 surgery (18 transplanted layers + 6 quadratic anchor layers [3, 7, 11, 15, 19, 23]).
derek-thomas/ScienceQA)| Metric | Teacher Baseline (llava-interleave-qwen-0.5b) |
Pre-KD Zero-Shot (asvd_attention) |
Post-KD Distilled (asvd_attention) |
Delta ($\Delta$) |
|---|---|---|---|---|
| ScienceQA Visual Accuracy (%) | 63.75% |
31.25% |
42.50% |
+11.25% |
| Image-Conditioned Answer PPL | 2.45 |
8.71 |
2.94 |
-5.76 |
232 (Stage 1: 88 | Stage 2: 144)120.1s | 17071 MB on NVIDIA RTX 3090 24GB2026-11-11T04:59:52.878143Zattention_weights.safetensors (361.30 MB): Transplanted decoder attention & layernorm parameters (FP16; loads directly onto frozen base llava-hf/llava-interleave-qwen-0.5b-hf)processor & tokenizer configs + training_metrics.jsonimport torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoProcessor, LlavaForConditionalGeneration
from scripts.benchmarks.qwen_zoo_train_mix import apply_qwen_attention_surgery
repo_id = "tahamajs/llava-qwen-0.5b-asvd-attention-scienceqa"
processor = AutoProcessor.from_pretrained(repo_id)
model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-interleave-qwen-0.5b-hf", torch_dtype=torch.float16)
model.language_model = apply_qwen_attention_surgery(model.language_model, attn_key="asvd_attention", pattern="hybrid_3_1")
ckpt_path = hf_hub_download(repo_id=repo_id, filename="attention_weights.safetensors")
model.load_state_dict(load_file(ckpt_path), strict=False)
model.eval()
import torch
from neuralops.hub import load_surgeried_model
# Load model and tokenizer/processor in one line directly from Hugging Face Hub
model, tokenizer = load_surgeried_model("tahamajs/llava-qwen-0.5b-asvd-attention-scienceqa")
model.eval()
print(f"Successfully loaded tahamajs/llava-qwen-0.5b-asvd-attention-scienceqa with NeuralOps modular surgical weights!")
@software{neuralops2026vlmzoo,
title = {NeuralOps: Multimodal Vision-Language Subquadratic Attention Surgery Zoo (LLaVA-ScienceQA)},
author = {Majlesi, Taha},
year = {2026},
url = {https://github.com/tahamajs/NeuralOps},
note = {Hub Repository: tahamajs/llava-qwen-0.5b-asvd-attention-scienceqa}
}
llava-hf/llava-interleave-qwen-0.5b-hf