中文Sign up

legoskier/sft-dpo-qwen-078-plus

🤗 Hugging Face sourcetext-generationapache-2.0-other⚠ No SHA-256 weightsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo legoskier/sft-dpo-qwen-078-plus ./model-folder
Needs a seeder →

qwen3-4b-sft-dpo-combined-078

This model is a two-stage fine-tuned version of Qwen/Qwen3-4B-Instruct-2507:

Training Pipeline

Stage 1: SFT (Supervised Fine-Tuning)

  • Dataset: u-10bei/structured_data_with_cot_dataset_512_v2
  • Learning rate: 1e-6
  • Epochs: 3
  • LoRA: r=64, alpha=128
  • StructEval Score: 0.780 (baseline: 0.700)

Stage 2: DPO (Direct Preference Optimization)

  • Base: SFT model (0.780)
  • Dataset: u-10bei/dpo-dataset-qwen-cot
  • Learning rate: 5e-08
  • Epochs: 1
  • Beta: 0.1
  • LoRA: r=8, alpha=16

This repository contains the full-merged 16-bit weights (SFT + DPO combined).

Training Configuration

SFT Stage

Base model: Qwen/Qwen3-4B-Instruct-2507
Method: QLoRA (4-bit)
Max sequence length: 512
Epochs: 3
Learning rate: 1e-6
LoRA: r=64, alpha=128
GPU: A100 (BFloat16)
Result: StructEval score 0.780

DPO Stage

Base: SFT model (0.780)
Method: DPO with QLoRA
Max sequence length: 1024
Epochs: 1
Learning rate: 5e-08
Beta: 0.1
LoRA: r=8, alpha=16
GPU: A100 (BFloat16)

Usage

Since this is a merged model, you can use it directly:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "your_username/sft-dpo-qwen-078-plus"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Example inference
prompt = "Your question here"
messages = [{"role": "user", "content": prompt}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

outputs = model.generate(inputs, max_new_tokens=512, temperature=0.0)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Sources & License

Training Data

Compliance

Users must comply with:

  • Base model license (Qwen)
  • Dataset licenses (MIT)
  • All applicable terms of use

Performance

  • SFT only: StructEval score 0.780
  • SFT + DPO: [To be evaluated]

Expected improvement: +0.02 to +0.04 points