Qwen/Qwen2.5-0.5B (minimal_svd) — NeuralOps Qwen Attention Zoo
Module Summary (MinimalSVDAttention): MinimalSVD: SVD-Factorized Low-Rank Projection Attention. Hardened for multi-architecture deployment across DistilBERT, RoBERTa, GPT-2, and Qwen3.5.
This repository contains a distilled Qwen/Qwen2.5-0.5B checkpoint whose decoder attention blocks have undergone surgical replacement with minimal_svd from the NeuralOps ATTENTION_REGISTRY, trained on WikiText-2 (Salesforce/wikitext / wikitext-2-raw-v1).
Module Capability Profile
Surgery Topology
- Pattern:
hybrid_3_1
- Layer Layout: 3:1 interleaved hybrid (
hybrid_3_1): 18/24 layers transplanted with minimal_svd via QwenAttentionAdapter (14:2 GQA expansion, Single-o_proj contract), and 6/24 full quadratic anchor layers preserved at indices [3, 7, 11, 15, 19, 23].
- Adapter Contract:
QwenAttentionAdapter maps Qwen2Attention projections (14 query heads, 2 KV heads, head_dim=64, hidden_size=896) with automatic 7x KV-head expansion where required and enforces the Single-o_proj bypass contract.
2-Stage Knowledge Distillation Telemetry
Weight Artifacts
Quick Start
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from scripts.benchmarks.qwen_zoo_train_mix import apply_qwen_attention_surgery
base_model_id = "Qwen/Qwen2.5-0.5B"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype=torch.float16)
model = apply_qwen_attention_surgery(model, attn_key="minimal_svd", pattern="hybrid_3_1")
# Load transplanted attention + full weights from this checkpoint directory or Hub repo
# e.g. model = AutoModelForCausalLM.from_pretrained("tahamajs/qwen-2.5-0.5b-minimal-svd-wikitext2", torch_dtype=torch.float16)
inputs = tokenizer("NeuralOps subquadratic attention enables", return_tensors="pt")
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Quick Start (Universal NeuralOps Hub API)
import torch
from neuralops.hub import load_surgeried_model
# Load model and tokenizer/processor in one line directly from Hugging Face Hub
model, tokenizer = load_surgeried_model("tahamajs/qwen-2.5-0.5b-minimal-svd-wikitext2")
model.eval()
print(f"Successfully loaded tahamajs/qwen-2.5-0.5b-minimal-svd-wikitext2 with NeuralOps modular surgical weights!")