Qwen1.5-MoE-A2.7B-MidAlign (task = SQuAD + SNLI + MMLU)
LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B theo baseline MidAlign, Alternate
Training: step chan = task step (LM loss), step le = contrastive align step.
Alternate Training
- Task step (chan):
L_task = L_LM + lb_loss_coef * L_LB.L_LMchi tinh tren phan dap an (prompt bi mask), tren pool gop cua cac tap: squad, snli, mmlu.- squad: 87599 sample (doc 87599, bo 0)
- snli: 549367 sample (doc 549367, bo 0)
- mmlu: 87691 sample (doc 99842, bo 12151)
lb_loss_coef= 0.01,num_experts= 60,top_k= 4
- Align step (le): contrastive Eq.(1) MidAlign (oneway, negative trong mini-batch 4) giua mean-pooled hidden state cau tieng Anh va cau target tai layer 12 (block 0-indexed = 11 / 24 layer), temperature = 1.5.
- Moi epoch: 5662 task step + 5662 align step. TASK la anchor: moi epoch dung HET 724657 sample task (SQuAD/SNLI/MMLU, English-only) >= 1 lan. Align = pool nho 121571 cap (nguon: flores, ntrex, ted; 500 cap/ngon ngu, boc ngau nhien seed 42), lap lai de du 724736 cap align / epoch (dung nhu paper).
- Contrastive: effective batch 16/rank, mini-batch 4, global_negatives = True, mask_false_negatives = True, lang_balance = True.
- world_size = 8, align_batch_size = 16 (per-rank, anchor), task_batch_size = 16 (per-rank), task_micro_batch_tokens = 4096.
LoRA
- Range layer
[8, 16)(0-indexed), attention / router / experts + shared expert. - Rank: attention 8, router 8, experts 8; alpha = 16, dropout = 0.1.
Training
- 3 epoch, gradient all-reduce thu cong (khong DDP), checkpoint chi giu ban moi nhat.
- Diagnostics (log theo tung step:
diagnostics/loss_log.jsonl):