🧠 GLM-4.5-Air-HS (QLoRA Fine-Tuned on Hyperswitch Corpus)
This repository contains QLoRA fine-tuning checkpoints for the model GLM-4.5-Air, trained on the Hyperswitch Rust repository.
🗂️ Repository Structure
GLM-4.5-Air-HS/
├── checkpoints/
│ ├── checkpoint-500/
│ ├── checkpoint-1000/
│ ├── ...
│ ├── checkpoint-7000/
│ └── final-checkpoint/ # step 7800
└── Final Model/ # merged adapter (coming soon)
⚙️ Training Overview
| Step | Eval Loss | Perplexity | Notes |
|---|---|---|---|
| 0 | 2.82 | 16.78 | Baseline before training |
| 1000 | 2.06 | 7.82 | Early stabilization |
| 2000 | 1.37 | 3.95 | Major improvement |
| 3000 | 1.21 | 3.35 | Smooth convergence |
| 4000 | 1.15 | 3.16 | Stable |
| 5000 | 1.06 | 2.89 | Optimal zone begins |
| 6000 | 0.93 | 2.52 | Strong generalization |
| 7000 | 0.88 | 2.41 | Best overall checkpoint |
✅ Observation: Continuous, monotonic improvement in loss and perplexity across steps suggests stable QLoRA convergence without overfitting.
🧮 Configuration Summary
# GLM-4.5-Air QLoRA Training Configuration
optimization:
micro_batch_per_gpu: 1
grad_accum_steps: 16
learning_rate: 5e-6
weight_decay: 0.01
warmup_ratio: 0.08
betas: [0.9, 0.95]
eps: 1e-8
lora:
r: 64
lora_alpha: 128
target_modules: ["q_proj", "k_proj", "v_proj", "o_proj"]
lora_dropout: 0.05
bias: "none"
quantization:
load_in_4bit: true
bnb_4bit_compute_dtype: "float32"
bnb_4bit_use_double_quant: true
bnb_4bit_quant_type: "nf4"
logging:
log_interval: 10
eval_interval: 1000
checkpoint_interval: 1000
max_checkpoints: 8
memory:
use_cpu_offload: true
use_deepspeed_zero3: true
gradient_checkpointing: true
use_cache: false
🧩 Model Objective
- Base Model: GLM-4.5-Air
- Fine-Tuning Mode: QLoRA (4-bit, rank 64)
- Data: 12,000 training + 1,500 validation datapoints
- Precision: NF4 quantization with double quant + 4bit compute (float32)
- Optimizer: AdamW (β₁=0.9, β₂=0.95, ε=1e-8)
QLoRA modifies only ~0.1% of parameters, so catastrophic forgetting is rare.
Maintained by: Juspay AI Research Contributor: Mynampati Sri Ranganadha Avinash (@ashx098)