This repository contains a surgically transplanted and distilled checkpoint of Qwen/Qwen2.5-0.5B where 18 out of 24 decoder attention blocks (3:1 hybrid layout) have been replaced with the asvd_attention (ASVDAttention) operator from the NeuralOps Library, followed by 2-stage hidden-representation knowledge distillation on WikiText-2 (Salesforce/wikitext, wikitext-2-raw-v1).
ASVDAttention: Hardened ASVD Attention with Factored Low-Rank Matrix Decomposition, GQA, RoPE, and LoRA. Computational Complexity: - Factorized Projections: $\mathcal{O}(B \cdot N \cdot r \cdot (d + d_{\text{all}}))$ - Attention Scoring: $\mathcal{O}(B \cdot H \cdot N^2 \cdot d_k)$ Tensor Dimensions: - hidden_states: $[B, N, D]$ where $B$ is batch size, $N$ is sequence length, and $D$ is hidden dimension. - Output: Tuple of $([B, N, D], \text{None})$.
| Property | Specification |
|---|---|
Registry Key (ATTENTION_REGISTRY) |
asvd_attention |
| NeuralOps Class & Family | ASVDAttention (svd) |
| Source Implementation | neuralops/layers/svd/asvd_attention.py |
| Base Architecture | Qwen/Qwen2.5-0.5B (494M params, 24 layers, $D=896$, $H_q=14$, $H_{kv}=2$, $d_h=64$) |
| Causal Autoregressive Support | Yes |
| Grouped-Query Attention (GQA) Handling | Yes |
| Rotary Position Embedding (RoPE) | Adapter-Handled ($ heta = 1{,}000{,}000$) |
| 3-Level Kernel Dispatch | CUDA / Triton / PyTorch 3-Level |
hybrid_3_1)To balance subquadratic/linear-complexity efficiency against full induction-head recall in compact 0.5B causal language models, NeuralOps applies a deterministic hybrid_3_1 surgery ratio across the 24 Qwen2 decoder blocks:
75%): Layer indices [0, 1, 2, 4, 5, 6, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 21, 22] use asvd_attention wrapped in QwenAttentionAdapter.25%): Layer indices [3, 7, 11, 15, 19, 23] preserve exact pretrained Qwen2 Grouped-Query Softmax Attention to maintain global contextual routing and retrieval anchors.o_proj Weight Inheritance: Each transplanted layer initializes from the teacher's exact pretrained q_proj, k_proj, v_proj, and o_proj tensors (with zero duplicated output projections and GQA KV-head expansion from 2 -> 14 heads), avoiding uninitialized projection collapse.Empirical Verification Guarantee: Distillation trained via the NeuralOps Offline Representation Caching Architecture (§21.1) on an NVIDIA GeForce RTX 3090 (24GB) with 0 MB Teacher GPU VRAM and deterministic weight serialization. & Distillation Telemetry
All metrics below are directly measured on the WikiText-2 (wikitext-2-raw-v1) validation split at sequence length 512 (batch_size=8, grad_accum=4) on physical cluster hardware (NVIDIA GeForce RTX 3090 24GB):
| Metric | Value | Notes |
|---|---|---|
Teacher Baseline (Qwen/Qwen2.5-0.5B) |
15.38 |
Reference exact GQA Softmax causal perplexity |
| Pre-KD Zero-Shot Student PPL | 254,182.30 |
Immediately after 18/24 layer surgical transplant |
| Post-KD Distilled Student PPL | 77.30 |
After 2-stage hidden-state representation distillation |
| Absolute Perplexity Delta ($\Delta$) | -254,105.00 |
Perplexity change achieved via distillation |
| Relative PPL Reduction | 99.97% |
Percentage recovery from zero-shot post-surgery spike |
30% budget, 124 steps):self_attn) and layer normalizations (layernorm) only; embeddings, MLPs, and lm_head stay frozen.GradScaler, AdamW (lr=2.0e-4, weight_decay=0.01, gradient clipping 1.0), minimizing layer-wise hidden-state $ ext{MSE} + 0.25 \cdot (1 - \cos(\mathbf{h}_s, \mathbf{h}_t))$.70% budget, 283 steps):lr=6.0e-5 across all 24 transformer residual streams (lm_head bypassed to maximize throughput).Salesforce/wikitext (wikitext-2-raw-v1, seq_len=512)407 (Stage 1: 124 | Stage 2: 283)180.4s19022 MB2026-11-10T00:26:54.593499ZThis repository ships both standalone full-model weights and modular parameter-efficient adapter weights in FP16:
| Artifact | Size | Description |
|---|---|---|
model.safetensors |
922.76 MB |
Complete standalone Qwen2 model weights (FP16) |
attention_weights.safetensors |
64.55 MB |
Transplanted attention adapter & projection weights only (FP16) |
config.json |
< 4 KB |
Architecture config tagged with "attention_type": "asvd_attention" |
training_metrics.json |
< 1 KB |
Verifiable machine-readable telemetry & PPL log |
Tokenizer (tokenizer.json, vocab.json, merges.txt) |
~15 MB |
Standard Qwen2.5 BPE tokenizer assets |
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForCausalLM, AutoTokenizer
from neuralops.core.registry import create_attention
from neuralops.layers.adapters.qwen_adapter import QwenAttentionAdapter
repo_id = "tahamajs/qwen-2.5-0.5b-asvd-attention-wikitext2"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", torch_dtype=torch.float16)
# Re-apply hybrid_3_1 surgery onto 18/24 decoder layers
hidden_size = model.config.hidden_size
num_heads = model.config.num_attention_heads
num_kv_heads = getattr(model.config, "num_key_value_heads", num_heads)
head_dim = getattr(model.config, "head_dim", hidden_size // num_heads)
for idx, layer in enumerate(model.model.layers):
if idx % 4 == 3:
continue
core_attn = create_attention("asvd_attention", dim=hidden_size, num_heads=num_heads, config=model.config)
layer.self_attn = QwenAttentionAdapter(
core_attention=core_attn,
embed_dim=hidden_size,
num_heads=num_heads,
num_kv_heads=num_kv_heads,
head_dim=head_dim,
inherit_weights_from=layer.self_attn,
is_causal=True,
bypass_internal_out_proj=True,
)
# Load distilled attention parameters from Hugging Face Hub
weights_path = hf_hub_download(repo_id=repo_id, filename="attention_weights.safetensors")
model.load_state_dict(load_file(weights_path), strict=False)
model = model.to("cuda" if torch.cuda.is_available() else "cpu").eval()
prompt = "Subquadratic attention surgery in large language models enables"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.7)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Because model.safetensors and attention_weights.safetensors are both tracked in this repository, you can inspect all distilled parameters directly:
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
state = load_file(hf_hub_download("tahamajs/qwen-2.5-0.5b-asvd-attention-wikitext2", "attention_weights.safetensors"))
print(f"Loaded {len(state)} transplanted attention parameter tensors.")
import torch
from neuralops.hub import load_surgeried_model
# Load model and tokenizer/processor in one line directly from Hugging Face Hub
model, tokenizer = load_surgeried_model("tahamajs/qwen-2.5-0.5b-asvd-attention-wikitext2")
model.eval()
print(f"Successfully loaded tahamajs/qwen-2.5-0.5b-asvd-attention-wikitext2 with NeuralOps modular surgical weights!")
@software{neuralops2026qwenzoo,
title = {NeuralOps: Subquadratic & Hybrid Attention Surgery Zoo for Qwen-2.5-0.5B},
author = {Majlesi, Taha},
year = {2026},
url = {https://github.com/tahamajs/NeuralOps}
}
model.safetensors 968 MBattention_weights.safetensors 68 MBQwen/Qwen2.5-0.5B