GalerkinAttention: GalerkinAttention neural operator. Galerkin Attention: Petrov-Galerkin Projection Attention.
This repository packages a modular NeuralOps LLaVA-Qwen-0.5B Multimodal Attention Surgery Zoo checkpoint derived from llava-hf/llava-interleave-qwen-0.5b-hf using galerkin_attention and evaluated on ScienceQA (Image-Bearing STEM Subset, validation).
| Property | Specification |
|---|---|
| Zoo Track | llava_vlm (LLaVA-Qwen-0.5B Multimodal Attention Surgery Zoo) |
| Operator / Registry Key | galerkin_attention (GalerkinAttention, family linear) |
| Source Implementation | neuralops/layers/linear/galerkin_attention.py |
| Base Architecture | llava-hf/llava-interleave-qwen-0.5b-hf — Frozen SigLIP Vision Tower + 2-layer MLP Projector + 24-layer Qwen2 decoder (D=1024, H=16, head_dim=64) |
| Surgical / Compression Scope | 18 / 24 decoder/encoder layers (hybrid_3_1) |
| Modified & Saved Components | 18/24 decoder language_model.model.layers.{i}.self_attn + trained decoder layernorms |
| Frozen & Excluded Components | Frozen SigLIP vision_tower.* (27 layers), multi_modal_projector.*, 6 quadratic anchor layers [3, 7, 11, 15, 19, 23], decoder MLPs & embeddings |
Empirical Verification Guarantee: Distillation trained via the NeuralOps Offline Representation Caching Architecture (§21.1) on an NVIDIA GeForce RTX 3090 (24GB) with 0 MB Teacher GPU VRAM and deterministic weight serialization.
| Metric | Teacher Baseline | Pre-KD / Zero-Shot | Post-KD / Final | Delta ($\Delta$) |
|---|---|---|---|---|
| Accuracy (%) | 63.75% |
32.50% |
62.50% |
+30.00% |
| Perplexity (PPL) | 2.45 |
5.12 |
2.10 |
-3.02 |
238 (Stage 1: 91 | Stage 2: 147)120.5s17788 MB2026-11-11T06:17:04.343177Zattention_weights.safetensors147.61 MB (154,778,944 bytes) | Total Package Size: 162.80 MBllava-hf/llava-interleave-qwen-0.5b-hf, 1.66 GB): 1.52 GB saved (-91.3%, 11.53x smaller artifact)| File on Disk | Size | Bytes | Role |
|---|---|---|---|
attention_weights.safetensors |
147.61 MB |
154,778,944 |
Modular surgical / compressed delta weights (FP16 safetensors) |
surgical_manifest.json |
50.6 KB |
51,800 |
Surgical / compression topology and modified-prefix manifest |
config.json |
6.4 KB |
6,577 |
Model architecture and NeuralOps surgery/compression metadata |
training_metrics.json |
765 B |
765 |
Machine-readable evaluation, telemetry, and distillation metrics |
added_tokens.json |
101 B |
101 |
Additional vocabulary tokens |
chat_template.json |
700 B |
700 |
Chat formatting template |
merges.txt |
1.59 MB |
1,671,853 |
BPE merge table |
preprocessor_config.json |
393 B |
393 |
Vision-language image preprocessor configuration |
processor_config.json |
134 B |
134 |
Checkpoint / tokenizer metadata |
special_tokens_map.json |
367 B |
367 |
Special token mappings |
tokenizer.json |
10.89 MB |
11,418,450 |
Fast tokenizer vocabulary and merges |
tokenizer_config.json |
1.6 KB |
1,602 |
Tokenizer special tokens and configuration |
vocab.json |
2.65 MB |
2,776,833 |
BPE vocabulary mapping |
import torch
from PIL import Image
from neuralops.hub import load_surgeried_model
model, processor = load_surgeried_model("tahamajs/llava-qwen-0.5b-galerkin-attention-scienceqa")
model.eval()
image = Image.new("RGB", (384, 384), color=(255, 255, 255))
inputs = processor(
text="<image>\nQuestion: What is shown in the figure?\nAnswer with the option letter:",
images=image,
return_tensors="pt",
)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=16)
print(processor.decode(outputs[0], skip_special_tokens=True))
Licensed under the MIT License (LICENSE).
@software{neuralops2026vlmzoo,
title = {NeuralOps: Multimodal Vision-Language Subquadratic Attention Surgery Zoo (LLaVA-ScienceQA)},
author = {Majlesi, Taha},
year = {2026},
url = {https://github.com/tahamajs/NeuralOps},
note = {Hub Repository: tahamajs/llava-qwen-0.5b-galerkin-attention-scienceqa}
}