GPT-2 Large — ASVD rank=640 + LoRA (CF-init)

LoRA adapter for GPT-2 Large after ASVD (activation-aware SVD) weight compression at rank=640, with closed-form residual initialization (LoftQ-style). The compressed + recovered model surpasses the uncompressed baseline while retaining a 1.6× theoretical inference speedup.

Results

13.7% better perplexity than the dense baseline, with 1.6× faster inference.

Method

ASVD compression (arXiv:2312.05821): weight columns are scaled by input activation statistics before SVD, then de-scaled after. This minimizes output error ||WX - W_k X|| rather than weight error ||W - W_k||, giving 34× better perplexity than naive SVD at the same rank.

Closed-form LoRA initialization: the same SVD that computes the ASVD approximation (using singular vectors [0:640]) also gives the best rank-16 correction of the residual via the next 16 singular vectors [640:656]. These are used to initialize LoRA weights before fine-tuning — placing the optimizer directly in the right subspace. This gives ~3% better final perplexity vs zero-init LoRA at identical training cost (55 min on a T4 GPU).

Key insight: the factored ASVD weights (A·B, rank=640) are used for inference — LoRA is discarded after training. The 1.6× speedup is fully preserved.

How to use

This adapter requires the ASVD-compressed base weights (not the original dense GPT-2 Large). You need to apply ASVD compression first, then load this adapter on top.

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer
from peft import PeftModel

# Step 1: load and compress base model with ASVD rank=640
# (see experiments/08_gpt2large_lora_cf_init/modal_experiment.py for the full compression code)

# Step 2: load the LoRA adapter on the compressed model
model = GPT2LMHeadModel.from_pretrained("gpt2-large")
# ... apply ASVD compression here ...
model = PeftModel.from_pretrained(model, "medelharchaoui/gpt2-large-asvd640-lora-cf-init")

tokenizer = GPT2Tokenizer.from_pretrained("gpt2-large")
inputs = tokenizer("The quick brown fox", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

Training details

Related experiments

This checkpoint is from Experiment 08 of the OptIAI research project, which investigates low-rank factorization of LLM weight matrices for inference speedup.

Framework versions

• PEFT 0.19.1
• Transformers 4.x