Qwen3-8B-LoRA-SFT

This repository contains a LoRA (Low-Rank Adaptation) adapter for Qwen3-8B-Base, fine-tuned using LLaMA Factory.

Model Description

• Base Model: Qwen/Qwen3-8B-Base
• Method: LoRA Fine-tuning (SFT)
• Language: Chinese / Japanese
• Training Date: February 28, 2026

Training Hyperparameters

The following hyperparameters were used during training:

• Learning Rate: 1e-4
• LRScheduler: Cosine
• Epochs: 5.0 (Completed ~3.6)
• Per Device Batch Size: 2
• Gradient Accumulation Steps: 4
• Optimizer: AdamW
• Mixed Precision: BF16
• Cutoff Length: 128
• LoRA Rank (r): 32
• LoRA Alpha: 32
• LoRA Dropout: 0.05
• LoRA Target Modules: all-linear

Training Results

The model underwent significant training with the following final metrics:

Loss Curve

[!NOTE] Training was configured with early_stopping_steps: 10. The process stopped at approximately 3.6 epochs as the eval_loss stabilized.

Evaluation Metrics (Test Set)

Post-training evaluation on the test set yielded the following scores:

• BLEU (raw): 21.36
• BLEU (Strict): 24.27
• chrF (raw): 57.90
• chrF (Strict): 61.49
• COMET: 0.7301
• Average Response Length (Chars): 8.0

Usage

You can use this adapter with the peft and transformers libraries:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_model_path = "Qwen/Qwen3-8B-Base"
adapter_path = "enet45/qwen3-8b-ja2zh-lora-v1.1"

tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter_path)

inputs = tokenizer("Introduce yourself in Chinese.", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations and Bias

This model is a LoRA adapter. It inherits the risks and biases of the base Qwen3-8B model. Users should perform their own safety testing before deploying in production environments.