Qwen/Qwen2.5-0.5B (rpc_attention) — NeuralOps Qwen Attention Zoo
Module Summary (RPCAttention): RPCAttention neural operator. Random Projection Compressed (RPC) Attention.
This repository contains a distilled Qwen/Qwen2.5-0.5B checkpoint whose decoder attention blocks have undergone surgical replacement with rpc_attention from the NeuralOps ATTENTION_REGISTRY, trained on WikiText-2 (Salesforce/wikitext / wikitext-2-raw-v1).
Module Capability Profile
Surgery Topology
- Pattern:
hybrid_3_1
- Layer Layout: 3:1 interleaved hybrid (
hybrid_3_1): 18/24 layers transplanted with rpc_attention via QwenAttentionAdapter (14:2 GQA expansion, Single-o_proj contract), and 6/24 full quadratic anchor layers preserved at indices [3, 7, 11, 15, 19, 23].
- Adapter Contract:
QwenAttentionAdapter maps Qwen2Attention projections (14 query heads, 2 KV heads, head_dim=64, hidden_size=896) with automatic 7x KV-head expansion where required and enforces the Single-o_proj bypass contract.
2-Stage Knowledge Distillation Telemetry
Weight Artifacts
Quick Start (Attention-Only Weights)
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForCausalLM, AutoTokenizer
from scripts.benchmarks.qwen_zoo_train_mix import apply_qwen_attention_surgery
base_model_id = "Qwen/Qwen2.5-0.5B"
repo_id = "tahamajs/qwen-2.5-0.5b-rpc-attention-wikitext2"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype=torch.float16)
model = apply_qwen_attention_surgery(model, attn_key="rpc_attention", pattern="hybrid_3_1")
ckpt_path = hf_hub_download(repo_id=repo_id, filename="attention_weights.safetensors")
model.load_state_dict(load_file(ckpt_path), strict=False)
model = model.eval()
inputs = tokenizer("NeuralOps subquadratic attention enables", return_tensors="pt")
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Quick Start (Universal NeuralOps Hub API)
import torch
from neuralops.hub import load_surgeried_model
# Load model and tokenizer/processor in one line directly from Hugging Face Hub
model, tokenizer = load_surgeried_model("tahamajs/qwen-2.5-0.5b-rpc-attention-wikitext2")
model.eval()
print(f"Successfully loaded tahamajs/qwen-2.5-0.5b-rpc-attention-wikitext2 with NeuralOps modular surgical weights!")