Qwen1.5-MoE-A2.7B-SNLI-Task-Only

Day la LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B, mot mo hinh Mixture-of-Experts, tren task SNLI (Stanford Natural Language Inference) — phan loai quan he giua premise/hypothesis thanh entailment / neutral / contradiction.

Prompt format

Premise: <premise>
Hypothesis: <hypothesis>
Question: What is the relationship between the premise and the hypothesis? Choose one: entailment, neutral, or contradiction.
Answer: <entailment|neutral|contradiction>

Cau hinh LoRA

• Layer duoc finetune: [8, 16) trong tong so 24 layer (tuong ung khoang 1L/3 -> 2L/3), giong het setting cua code alignment finetuning.
• Module duoc gan LoRA: attention, router, experts trong khoang layer tren, moi thanh phan mot rank rieng qua rank_pattern cua PEFT:
  • attention: r = 16
  • router: r = 4
  • experts: r = 16
• alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

• L_LM: cross-entropy chuan tren token tiep theo, CHI tinh tren phan <label_word><eos> (prompt bi mask, giong instruction-tuning/SFT tieu chuan).
• L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
• lb_loss_coef (lambda) = 0.01
• num_experts = 60, top_k = 4

Du lieu

SNLI, doc tu data/english_task/snli/train.json (list object {"premise", "hypothesis", "label"}, label 0/1/2 = entailment/neutral/contradiction).

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png, diagnostics/loss_curve_smoothed.png va diagnostics/accuracy_curve.png (do chinh xac xap xi tren token dau tien cua nhan, tinh tren batch train, da lam min moi 50 step).