Qwen3-4B-LoRA-SFT

This repository contains a LoRA (Low-Rank Adaptation) adapter for Qwen3-4B-Instruct-2507, fine-tuned using LLaMA Factory.

Model Description

• Base Model: Qwen/Qwen3-4B-Instruct-2507
• Method: LoRA Fine-tuning (SFT)
• Language: Chinese / Japanese
• Training Date: February 28, 2026

Training Hyperparameters

The following hyperparameters were used during training:

• Learning Rate: 3e-5
• LRScheduler: Cosine
• Epochs: 5.0
• Per Device Batch Size: 8
• Gradient Accumulation Steps: 4
• Optimizer: AdamW
• Mixed Precision: BF16
• Cutoff Length: 128
• LoRA Rank (r): 32
• LoRA Alpha: 32
• LoRA Dropout: 0.05
• LoRA Target Modules: all-linear

Training Results

The model underwent significant training with the following final metrics:

Evaluation Metrics (Test Set)

Post-training evaluation on the test set yielded the following scores:

• BLEU (raw): 36.89
• BLEU (Strict): 42.73
• chrF (raw): 69.98
• chrF (Strict): 74.23
• COMET: 0.6873
• Average Response Length (Chars): 14.3

Usage

You can use this adapter with the peft and transformers libraries:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_model_path = "Qwen/Qwen3-8B-Base"
adapter_path = "enet45/qwen3-4b-ja2zh-lora-v1.1"

tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter_path)

inputs = tokenizer("Introduce yourself in Chinese.", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations and Bias

This model is a LoRA adapter. It inherits the risks and biases of the base Qwen3-4B-Instruct-2507 model. Users should perform their own safety testing before deploying in production environments.