Qwen3 0.6B Base LoRA SFT

This repository contains LoRA adapter weights fine-tuned from Qwen/Qwen3-0.6B-Base with TRL SFTTrainer and PEFT LoRA.

The repository is an adapter repository, not a standalone full model. Load it together with the base model.

Training and Evaluation Data

Dataset: trl-lib/Capybara

This run uses a conversational instruction-tuning dataset in messages format:

{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Splits used in this run:

The source text was lightly normalized by trimming whitespace, merging repeated whitespace, mapping common role names such as human/gpt to user/assistant, and filtering rows without both a user and assistant message.

Training Procedure

• Base model: Qwen/Qwen3-0.6B-Base
• Method: supervised fine-tuning with LoRA adapters
• Trainer: TRL SFTTrainer
• PEFT method: LoRA
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Max length: 1024
• Train batch size per device: 1
• Gradient accumulation steps: 8
• Effective batch size: 8
• Learning rate: 0.0002
• Epochs: 2.0
• Weight decay: 0.01
• Warmup ratio: 0.03
• Eval steps: 2500
• Save steps: 2500
• Assistant-only loss: True
• Packing: False

Evaluation Results

• train_train_runtime: 5267.8313
• train_train_samples_per_second: 3.797
• train_train_steps_per_second: 0.475
• train_total_flos: 3.825816158208e+16
• train_train_loss: 1.1246953674316407
• validation_loss: 1.1391483545303345
• validation_runtime: 2.613
• validation_samples_per_second: 38.271
• validation_steps_per_second: 19.135
• test_loss: 1.2022475004196167
• test_runtime: 2.5761
• test_samples_per_second: 38.819
• test_steps_per_second: 19.409
• validation_perplexity: 3.124106600673977
• test_perplexity: 3.3275872785829663

Evaluation loss is token-level cross entropy on held-out conversations. Perplexity is computed as exp(loss) when the loss is in a numerically safe range.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "Qwen/Qwen3-0.6B-Base"
adapter_id = "JumpHigh/qwen3-0.6b-base-lora-sft"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [{"role": "user", "content": "Explain LoRA in simple terms."}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=160, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Intended Use and Limitations

This is a small LoRA SFT adapter for instruction-following experiments. It is useful for demonstrating the fine-tuning workflow and comparing LoRA with full fine-tuning. It should be evaluated carefully before production use, especially for factuality, safety, multilingual behavior, and domain-specific correctness.

Framework versions

• PEFT 0.19.1