mostwanted002/Sage-A1.5-Ablitrex-abliterated-BF16

Benchmark Comparison: Sage-A1.5 vs. Sage-A1.5 Abliterated (Abliterix)

Abliterated with Abliterix (directional ablation, LoRA-Δ + direct projection). Evaluated with lm-evaluation-harness (vLLM backend, bf16, 0-shot, seed=0).

Summary

Metric	Value
Mean capability drop (orig − ablit)	3.63 pts
Worst single-task drop	`mmlu_machine_learning` (−15.18 pts)
Best retained task	`mmlu_high_school_computer_science` (+2.00 pts)
Tasks evaluated	62 (MMLU + subcategories, ARC-C, HellaSwag, Winogrande)

Full Results

Task	Original	Abliterated	Δ (pts)
arc_challenge	51.71	48.89	−2.82
hellaswag	61.71	55.50	−6.21
mmlu	82.50	78.31	−4.19
winogrande	73.40	69.61	−3.79

mmlu_abstract_algebra	64.00	61.00	−3.00
mmlu_anatomy	85.19	82.96	−2.22
mmlu_astronomy	93.42	91.45	−1.97
mmlu_business_ethics	84.00	81.00	−3.00
mmlu_clinical_knowledge	88.30	84.15	−4.15
mmlu_college_biology	93.06	93.06	0.00
mmlu_college_chemistry	61.00	60.00	−1.00
mmlu_college_computer_science	74.00	73.00	−1.00
mmlu_college_mathematics	67.00	57.00	−10.00
mmlu_college_medicine	84.39	80.93	−3.47
mmlu_college_physics	67.65	65.69	−1.96
mmlu_computer_security	85.00	81.00	−4.00
mmlu_conceptual_physics	94.04	91.92	−2.13
mmlu_econometrics	78.95	73.68	−5.26
mmlu_electrical_engineering	86.21	83.45	−2.76
mmlu_elementary_mathematics	79.89	78.31	−1.59
mmlu_formal_logic	71.43	65.87	−5.56
mmlu_global_facts	52.00	49.00	−3.00
mmlu_high_school_biology	93.87	92.90	−0.97
mmlu_high_school_chemistry	80.30	74.88	−5.42
mmlu_high_school_computer_science	88.00	90.00	+2.00
mmlu_high_school_european_history	83.64	82.42	−1.21
mmlu_high_school_geography	93.43	89.90	−3.54
mmlu_high_school_government_and_politics	98.45	94.82	−3.63
mmlu_high_school_macroeconomics	89.74	86.41	−3.33
mmlu_high_school_mathematics	59.63	54.07	−5.56
mmlu_high_school_microeconomics	94.96	92.86	−2.10
mmlu_high_school_physics	82.78	74.17	−8.61
mmlu_high_school_psychology	95.41	94.68	−0.73
mmlu_high_school_statistics	79.17	76.85	−2.32
mmlu_high_school_us_history	93.14	91.67	−1.47
mmlu_high_school_world_history	91.14	90.72	−0.42
mmlu_human_aging	82.51	78.03	−4.48
mmlu_human_sexuality	89.31	83.97	−5.34
mmlu_humanities	76.09	70.33	−5.76
mmlu_international_law	91.74	88.43	−3.31
mmlu_jurisprudence	91.67	84.26	−7.41
mmlu_logical_fallacies	88.96	85.89	−3.07
mmlu_machine_learning	78.57	63.39	−15.18
mmlu_management	90.29	88.35	−1.94
mmlu_marketing	93.16	93.16	0.00
mmlu_medical_genetics	96.00	93.00	−3.00
mmlu_miscellaneous	94.64	92.72	−1.92
mmlu_moral_disputes	84.39	81.79	−2.60
mmlu_moral_scenarios	61.90	49.27	−12.63
mmlu_nutrition	89.54	86.93	−2.61
mmlu_other	86.06	83.14	−2.93
mmlu_philosophy	86.82	84.24	−2.57
mmlu_prehistory	90.74	86.42	−4.32
mmlu_professional_accounting	71.99	67.73	−4.26
mmlu_professional_law	67.01	60.76	−6.26
mmlu_professional_medicine	93.38	88.60	−4.78
mmlu_professional_psychology	87.91	79.90	−8.01
mmlu_public_relations	72.73	71.82	−0.91
mmlu_security_studies	83.67	77.55	−6.12
mmlu_social_sciences	90.48	86.51	−3.97
mmlu_sociology	93.04	90.55	−2.49
mmlu_stem	80.75	77.45	−3.30
mmlu_us_foreign_policy	96.00	93.00	−3.00
mmlu_virology	57.23	54.22	−3.01
mmlu_world_religions	89.47	88.89	−0.59

Notable Deltas

Direction	Task	Δ	Interpretation
Worst drop	mmlu_machine_learning	−15.18 pts	Heavy reasoning + domain-specific math; abliteration disrupted the reasoning chain most here
2nd worst	mmlu_moral_scenarios	−12.63 pts	Morally-ambiguous classification — overlaps with the refusal direction itself
3rd worst	mmlu_college_mathematics	−10.00 pts	Formal math reasoning
Best retained	mmlu_high_school_computer_science	+2.00 pts	Factual/recall-heavy; least coupled to the abliterated direction
No change	mmlu_college_biology	0.00	Identical scores
No change	mmlu_marketing	0.00	Identical scores

mostwanted002/Sage-A1.5-Ablitrex-abliterated-BF16

Benchmark Comparison: Sage-A1.5 vs. Sage-A1.5 Abliterated (Abliterix)

Summary

Full Results

Notable Deltas

为此模型做种