What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation

Code and raw experimental results for the paper:

What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation Babak Barazandeh Cribl AI Research Lab arXiv:2609.32002

Overview

LoRA's rank hyperparameter is widely treated as a capacity knob — higher rank, richer function class. This paper shows that under per-factor Frobenius norm budgets (the regime induced by weight decay), the reachable set of the product BA is rank-free: its complexity depends on the budget product ρ = B_B · B_A, not on the nominal rank r. Rank controls only affordability — whether the model can spend the nuclear-norm budget — not the richness of what it can express once it does.

The experiments below verify the paper's four main claims: (A) implicit low-nuclear-norm bias regardless of rank, (B) generalization gap flat in r when the budget binds (with plateau ∝ ρ), (B') worst-case gap attained by rank-1 matrices, (C) spectral threshold scaling r_spec(ε) under polynomial and geometric decay, and (D) complexity branches under per-factor vs. joint budgets. All experiments are synthetic, run on CPU, and reproduce exactly from fixed seeds.

Files

Installation

pip install -r requirements.txt

Everything runs on CPU. No GPU, no downloads, no API calls.

Quick start

Each experiment is self-contained. Run from the repo directory:

# Experiment A — implicit nuclear norm bias (~3 min)
python run_expA_implicit_bias.py

# Experiment B — gap vs. rank under hard budget (~2 min)
python run_expB_gap.py

# Experiment B replication — fresh seeds, m=d=64 (~1 min)
python run_expB_replication.py

# Experiment B' — worst-case gap maximization (~2 min)
python run_expBprime_worstcase.py

# Experiment B' init control — Kaiming-normal B (~2 min)
python run_expBprime_fullinit.py

# Experiment C — spectral threshold scaling (~5 min)
python run_expC_v2_threshold.py

# Experiment D — complexity branches (~1 min, no training)
python run_expD_complexity.py

All seeds are fixed — numbers reproduce exactly. The included results_*.json files let you regenerate figures without re-running training:

python reanalyze_expB.py     # → B1_gap_vs_rank_v2.png, B2_plateau_vs_rho_v2.png
python replot_bprime.py      # → Bprime_worst_case.png

Citation

@article{barazandeh2026rankbuy,
  title={What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation},
  author={Barazandeh, Babak},
  journal={arXiv preprint arXiv:2609.32002},
  year={2026}
}

License

Apache 2.0