This repository provides a multimodal Vision-Language Model (VLM) derived from llava-hf/llava-interleave-qwen-0.5b-hf (SigLIP vision encoder + MLP projector + 24-layer Qwen2 decoder, D=1024, H=16) where 18 / 24 decoder attention blocks (hybrid_3_1) have been replaced with block_transformer_hiba (BlockTransformerHIBA) and distilled on image-bearing questions from derek-thomas/ScienceQA.
BlockTransformerHIBA: Hierarchical block attention: * Intra-block: dense attention within each block * Inter-block: sparse top-k attention to non-local blocks * Combined with a SINGLE softmax over [intra_scores | inter_scores]
| Property | Specification |
|---|---|
Registry Key (ATTENTION_REGISTRY) |
block_transformer_hiba |
| NeuralOps Class & Family | BlockTransformerHIBA (adaptive) |
| Source Implementation | neuralops/layers/adaptive/block_transformer_hiba.py |
| Vision Tower (Frozen) | SigLIP Vision Encoder + 2-layer Multimodal MLP Projector |
| Language Decoder Geometry | 24 layers, D=1024, H=16, d_h=64 (exact power-of-2 widths) |
In LlavaForConditionalGeneration, visual patch tokens extracted by the frozen SigLIP vision tower are interleaved with text tokens inside language_model.model.layers. Keeping vision_tower and multi_modal_projector completely frozen isolates how well subquadratic mechanism block_transformer_hiba handles dense visual-to-text cross-modal routing under hybrid_3_1 surgery (18 transplanted layers + 6 quadratic anchor layers [3, 7, 11, 15, 19, 23]).
derek-thomas/ScienceQA)| Metric | Teacher Baseline (llava-interleave-qwen-0.5b) |
Pre-KD Zero-Shot (block_transformer_hiba) |
Post-KD Distilled (block_transformer_hiba) |
Delta ($\Delta$) |
|---|---|---|---|---|
| ScienceQA Visual Accuracy (%) | 63.75% |
41.25% |
42.50% |
+1.25% |
| Image-Conditioned Answer PPL | 2.45 |
2.90 |
2.87 |
-0.03 |
115 (Stage 1: 39 | Stage 2: 76)120.9s | 17321 MB on NVIDIA RTX 3090 24GB2026-11-11T05:08:18.402661Zattention_weights.safetensors (459.16 MB): Transplanted decoder attention & layernorm parameters (FP16; loads directly onto frozen base llava-hf/llava-interleave-qwen-0.5b-hf)processor & tokenizer configs + training_metrics.jsonimport torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoProcessor, LlavaForConditionalGeneration
from scripts.benchmarks.qwen_zoo_train_mix import apply_qwen_attention_surgery
repo_id = "tahamajs/llava-qwen-0.5b-block-transformer-hiba-scienceqa"
processor = AutoProcessor.from_pretrained(repo_id)
model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-interleave-qwen-0.5b-hf", torch_dtype=torch.float16)
model.language_model = apply_qwen_attention_surgery(model.language_model, attn_key="block_transformer_hiba", pattern="hybrid_3_1")
ckpt_path = hf_hub_download(repo_id=repo_id, filename="attention_weights.safetensors")
model.load_state_dict(load_file(ckpt_path), strict=False)
model.eval()
import torch
from neuralops.hub import load_surgeried_model
# Load model and tokenizer/processor in one line directly from Hugging Face Hub
model, tokenizer = load_surgeried_model("tahamajs/llava-qwen-0.5b-block-transformer-hiba-scienceqa")
model.eval()
print(f"Successfully loaded tahamajs/llava-qwen-0.5b-block-transformer-hiba-scienceqa with NeuralOps modular surgical weights!")
@software{neuralops2026vlmzoo,
title = {NeuralOps: Multimodal Vision-Language Subquadratic Attention Surgery Zoo (LLaVA-ScienceQA)},
author = {Majlesi, Taha},
year = {2026},
url = {https://github.com/tahamajs/NeuralOps},
note = {Hub Repository: tahamajs/llava-qwen-0.5b-block-transformer-hiba-scienceqa}
}