Gemma 4 E4B Fine-Tuned with Unsloth QLoRA

Competition: The Gemma 4 Good Hackathon on Kaggle
Tracks: Unsloth ($10K prize) + Impact Tracks
Framework: Unsloth — 2x faster fine-tuning
Base Model: google/gemma-4-e4b-it (4B params, instruction-tuned)

Highlights

• 99.6% training loss reduction — from 2.916 (baseline) to 0.0115 (final)
• 5 epochs of QLoRA fine-tuning on 10,000 high-quality samples
• Only 2.29% of parameters trained (146.8M / 6.4B) via rank-stabilized LoRA
• 12 hours total training on a single NVIDIA L4 GPU (24GB)

How to Use

With Unsloth (Recommended)

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    "bradduy/Any2AnyModels",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastModel.for_inference(model)

messages = [
    {"role": "user", "content": "Explain how renewable energy helps developing communities"}
]

inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

With Transformers + PEFT

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-e4b-it",
    device_map="auto",
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(base_model, "bradduy/Any2AnyModels")
tokenizer = AutoTokenizer.from_pretrained("bradduy/Any2AnyModels")

Training Details

Method

We used Unsloth's QLoRA implementation with rank-stabilized LoRA (RSLoRA) for parameter-efficient fine-tuning. The key innovation was discovering that multi-epoch training dramatically reduces loss with each additional pass over the data.

Configuration

Dataset

• Source: mlabonne/FineTome-100k
• Samples Used: 10,000 (first 10k)
• Format: Multi-turn chat conversations
• Chat Template: Gemma 4 native (role: "model", not "assistant")
• Masking: Only model responses contribute to loss (instruction tokens masked)

Hardware

• GPU: NVIDIA L4 (24GB VRAM)
• RAM: 32GB
• Training Time: ~12 hours (with checkpoint resume)
• GPU Memory Used: ~14.8GB during training

Experiment Journey

We ran 8 systematic experiments to find the optimal configuration:

The Multi-Epoch Discovery

The single most impactful finding: each additional epoch delivers a dramatic, consistent loss reduction:

Epoch 1: loss ~0.90  (learning the patterns)
Epoch 2: loss ~0.60  (reinforcing knowledge)
Epoch 3: loss ~0.30  (deep memorization)
Epoch 4: loss ~0.10  (fine polishing)
Epoch 5: loss ~0.01  (near-perfect fitting)

This pattern was consistent across experiments 06, 07, and 08. The loss drops happen at each epoch boundary as the model sees the training data again.

Other Key Insights

1. r=64 with RSLoRA is the sweet spot — r=128 converges slower and provides no benefit in multi-epoch settings
2. Lower LR (7e-5) stabilizes long training — higher LR (2e-4) causes instability after epoch 2
3. train_on_responses_only is essential — masks user/system tokens so the model only learns from responses
4. Checkpoint saving every 250 steps — long CUDA runs crash from memory fragmentation; resume from checkpoints solved this
5. 10k high-quality samples > 20k samples for multi-epoch — quality over quantity when doing multiple passes

Training Pipeline

Built entirely with Unsloth:

from unsloth import FastModel
from trl import SFTTrainer, SFTConfig
from unsloth.chat_templates import get_chat_template, train_on_responses_only

# 1. Load 4-bit quantized model
model, tokenizer = FastModel.from_pretrained(
    "unsloth/gemma-4-E4B-it-unsloth-bnb-4bit",
    max_seq_length=2048, load_in_4bit=True,
)

# 2. Apply LoRA adapters (r=64, RSLoRA)
model = FastModel.get_peft_model(model,
    finetune_vision_layers=False, finetune_language_layers=True,
    finetune_attention_modules=True, finetune_mlp_modules=True,
    r=64, lora_alpha=64, lora_dropout=0, bias="none",
    random_state=3407, use_rslora=True,
)

# 3. Setup Gemma 4 chat template
tokenizer = get_chat_template(tokenizer, chat_template="gemma-4")

# 4. Train with response-only masking
trainer = SFTTrainer(model=model, tokenizer=tokenizer, train_dataset=dataset,
    args=SFTConfig(
        per_device_train_batch_size=1, gradient_accumulation_steps=8,
        learning_rate=7e-5, num_train_epochs=5, lr_scheduler_type="cosine",
        warmup_steps=50, weight_decay=0.01, optim="adamw_8bit",
        save_strategy="steps", save_steps=250, save_total_limit=3,
    ),
)
trainer = train_on_responses_only(trainer,
    instruction_part="<|turn>user\n", response_part="<|turn>model\n",
)
trainer.train()

Reproduce Training

git clone https://github.com/bradduy/Any2AnyModels
cd Any2AnyModels
pip install unsloth

python scripts/train.py \
  --model unsloth/gemma-4-E4B-it-unsloth-bnb-4bit \
  --load-4bit --lora-rank 64 --use-rslora \
  --dataset mlabonne/FineTome-100k --max-samples 10000 \
  --num-epochs 5 --learning-rate 7e-5 --grad-accum 8 \
  --weight-decay 0.01 --warmup-steps 50 --scheduler cosine \
  --save-steps 250 --save-total-limit 3

Limitations

• Fine-tuned on English-only data (FineTome-100k)
• Optimized for instruction following, not domain-specific tasks
• 4B parameter model — larger models (26B, 31B) would perform better but require more VRAM
• Training loss ≠ downstream task performance; the model should be evaluated on specific benchmarks

Acknowledgments

• Google DeepMind for the Gemma 4 model family
• Unsloth for making QLoRA fine-tuning 2x faster and memory efficient
• Kaggle for hosting the Gemma 4 Good Hackathon
• mlabonne for the FineTome-100k dataset

License

Apache 2.0 (same as Gemma 4)