qwen2.5-0.5b-dpo-ultrafeedback

Qwen/Qwen2.5-0.5B-Instruct 以 DPO（Direct Preference Optimization） 在 HuggingFaceH4/ultrafeedback_binarized（train_prefs split）的 6000 組偏好資料對上做偏好對齊。

方法：DPO 是什麼？

DPO 直接用「同一個 prompt 的較好回答（chosen）與較差回答（rejected）」成對訓練， 不需要先訓練獨立的 reward model、也不需要 PPO 的強化學習迴圈。它把隱式獎勵定義為 policy 與 reference 模型的 log 機率比值：r(x, y) = beta * log(pi(y|x) / pi_ref(y|x))， 訓練目標是把 chosen 與 rejected 的隱式獎勵差距拉大。

beta = 0.1：控制 policy 可以偏離 reference 的程度——beta 越大越保守 （緊貼 reference），越小越激進（對齊訊號強、但容易偏離原模型導致輸出劣化）。

本 repo 存放的是 merged 模型：LoRA adapter（r=16）已用 merge_and_unload() 併回 base 權重（bfloat16），推論時不需要 PEFT，直接 from_pretrained 即可。

超參數

訓練框架：TRL DPOTrainer（loss_type=sigmoid）。資料前處理：套 Qwen chat template 轉為標準格式、過濾無明確偏好（score 平手／反轉）與過長樣本。

訓練曲線

最終 rewards/accuracies（滾動平均）：0.639

Before / After 對照（greedy decoding）

Prompt: What are the health benefits of intermittent fasting? Are there any risks I should know about?

原始模型

Intermittent fasting is a dietary approach that involves restricting food intake to only certain times of the day or week and then allowing yourself to eat normally during other times. This can help people lose weight and improve their overall health in several ways:

1. Weight loss: Intermittent fasting has been shown to be effective at helping people lose weight. It can lead to significant reductions in body mass index (BMI) and fat percentage.

2. Improved metabolism: By reducing calorie intake, intermittent fasting can increase your metabolic rate, which means you burn more calories throughout the day.

3. Reduced inflammation: Studies have found that intermittent fasting can reduce levels of inflammatory markers in the blood, which may help prevent chronic diseases like heart disease and diabetes.

4. Better sleep quality: Some studies suggest that intermittent fasting can improve sleep patterns and reduce symptoms of insomnia.

5. Increased energy levels: People who follow an intermittent fasting diet often report feeling more energized and less tired after eating.

However, it's important to note that intermittent fasting carries some risks:

1. Nutrient deficiencies: If not properly planned, intermittent fasting can lead to nutrient deficiencies, especially if you're not consuming enough calories from protein and healthy fats.

2. Hormonal changes: Some research suggests that intermittent fasting might

DPO 後模型

Intermittent fasting (IF) is a dietary regime that involves alternating between periods of eating and fasting. While it has gained popularity in recent years for its potential health benefits, including weight loss, improved insulin sensitivity, and reduced inflammation, there are also some concerns and risks associated with this approach.

Health Benefits of Intermittent Fasting

1. Weight Loss: Studies have shown that intermittent fasting can lead to significant weight loss compared to traditional calorie-controlled diets. This is because fasting triggers the release of hormones like ghrelin and leptin, which help regulate appetite and satiety.

2. Improved Insulin Sensitivity: Research indicates that intermittent fasting may improve insulin sensitivity, meaning your body's ability to use glucose more efficiently. This can be beneficial for people with type 2 diabetes or those at risk of developing it.

3. Reduced Risk of Chronic Diseases: Some studies suggest that intermittent fasting might reduce the risk of certain chronic diseases such as cardiovascular disease, type 2 diabetes, and some types of cancer.

4. Enhanced Metabolism: By reducing food intake during the fast period, intermittent fasting can increase metabolic rate, potentially leading to weight loss without significantly altering daily caloric intake.

5. Stress Reduction: Many

Prompt: Explain the difference between supervised and unsupervised learning to a high school student.

原始模型

Sure! Let's break down the differences between supervised and unsupervised learning in simple terms:

Supervised Learning

Supervised learning is like when you're trying to guess what color a picture is based on its features. Imagine you have a bunch of pictures of different types of flowers, and each one has a specific color. If I give you a new picture that looks similar to some of these flower pictures, you can use this information to figure out which type of flower it might be.

In supervised learning, we train our model (like a computer program) using labeled data. This means we provide it with examples of correct answers (the "labels") and then ask it to make guesses about what the answer should be for new, unseen examples. The goal is to learn patterns or relationships between the input features and the output labels.

For example:

• Classification: Suppose you want to classify images into two categories: cats or dogs.
• Regression: Predicting how long a car will travel based on its speed.

Unsupervised Learning

Unsupervised learning is like when you're trying to find hidden patterns in your data without any clear labels. It's great for finding groups within your data where there isn't an obvious way to tell them apart from others

DPO 後模型

Sure! Let's break down the differences between supervised and unsupervised learning in simple terms:

Supervised Learning

Supervised learning is like when you're trying to predict something based on what you already know or have data about it. Imagine you have a toy car that has wheels, but you don't know exactly how many wheels it has yet. With supervised learning, your computer learns from examples of cars with different numbers of wheels (like 4, 6, 8, etc.) and tries to figure out which number of wheels makes the most sense for this particular car.

• Example: If you have a dataset of cars with different numbers of wheels, your computer can learn to guess the correct number of wheels for any new car.

Unsupervised Learning

Unsupervised learning is like when you're trying to find patterns or groups within your data without knowing beforehand what those patterns are. Think of it as clustering - grouping similar items together, even if they don't look alike at first glance.

• Example: In a group of people who own different types of pets (cats, dogs, birds), your computer could use unsupervised learning to group them into categories such as "dogs," "birds," "cat owners," etc., just

Prompt: I have chicken, rice, and broccoli. Give me a simple 30-minute dinner recipe.

原始模型

Sure! Here's a simple 30-minute dinner recipe for you:

Chicken Rice Bowl

Ingredients:

• 1 whole chicken (about 2 pounds)
• 4 cups of high-quality brown rice
• 1 cup of frozen mixed vegetables (such as carrots, peas, corn, and bell peppers)
• 1/2 cup of chopped onion
• 1/2 cup of chopped garlic
• 1 tablespoon of olive oil
• Salt and pepper to taste
• Optional: 1/2 teaspoon of dried thyme or rosemary for garnish

Instructions:

1. Preheat the Oven: Preheat your oven to 375°F (190°C).

2. Cook the Chicken: Place the chicken in a large skillet over medium-high heat. Cook until the skin is crispy and the internal temperature reaches 165°F (74°C). Remove the chicken from the skillet and set it aside.

3. Prepare the Rice: In a large pot, combine the brown rice with the frozen vegetables. Bring to a boil, then reduce the heat to low, cover, and simmer for about 15 minutes, stirring occasionally, until the rice is tender but still slightly firm.

DPO 後模型

Certainly! Here's a simple and delicious 30-minute dinner recipe for you:

Chicken Rice Bowl

Ingredients:

• 1 cup cooked brown rice
• 2 tablespoons olive oil or vegetable oil
• 1 large onion, finely chopped
• 3 cloves garlic, minced
• 1 medium carrot, peeled and grated
• 1 small bell pepper, sliced
• 1 cup fresh spinach leaves (or any green leafy vegetables)
• 1/4 cup chopped fresh parsley
• Salt and pepper to taste
• 1 tablespoon lemon juice
• 1 teaspoon dried oregano
• 1 teaspoon dried basil
• 1/2 teaspoon red pepper flakes (optional)

Instructions:

1. Prepare the Vegetables: In a large skillet over medium heat, cook the onion and garlic until soft and fragrant. Add the grated carrot and bell pepper, and sauté for about 5 minutes.

2. Cook the Rice: In a large pot, bring the water to a boil. Add the cooked brown rice and reduce the heat to low. Cover the pot and let it simmer for about 15-20 minutes, or until the rice is tender but still slightly firm to the

Prompt: Write a short professional email asking my manager for two days off next week.

原始模型

Subject: Request for Two Days Off

Dear [Manager's Name],

I hope this message finds you well.

I am writing to request two days off from work on the following Monday and Tuesday of next week. I believe that taking time off will allow me to recharge and ensure that I can handle any additional responsibilities or commitments that may arise during this period.

Please let me know if there is anything specific I need to do in advance to make sure I have everything ready for when I return to work. Additionally, I would appreciate it if we could schedule a meeting with you to discuss the details of my request and any other relevant matters.

Thank you very much for your consideration. I look forward to hearing back from you soon.

Best regards,

[Your Full Name]
[Your Position]
[Your Contact Information]

DPO 後模型

Subject: Request for Two Days Off

Dear [Manager's Name],

I hope this message finds you well.

I am writing to request two days off from work on the following Monday and Tuesday of next week. I believe that taking time off will allow me to recharge, reflect on my responsibilities, and ensure that I can contribute effectively to our team during these periods.

The reason behind my request is as follows:

1. Personal Health: I have been feeling overwhelmed with work-related stress and need some time to relax and rejuvenate.
2. Team Dynamics: I believe it would be beneficial for us all to take a break from our daily tasks and focus on building stronger relationships within the team.
3. Professional Development: Taking a day off allows me to review my skills and knowledge, ensuring that I remain up-to-date with industry trends and best practices.

I understand that your schedule may be tight, but I assure you that I will make every effort to manage my time efficiently while still being available to assist you in any way possible.

Thank you very much for considering my request. I look forward to hearing back from you soon.

Best regards,

[Your Full Name]
[Your Position]
[Your Contact Information]

Prompt: What would happen if the Moon suddenly disappeared? Walk me through the consequences.

原始模型

If the Moon suddenly disappeared from our solar system, it would have profound and far-reaching effects on Earth and its inhabitants. Here's a detailed breakdown of what might happen:

1. Global Climate Change: The absence of the Moon would cause a significant shift in Earth's climate. Without the gravitational pull of the Moon, the planet would experience a drastic change in temperature patterns. This could lead to extreme weather events such as droughts, floods, and heatwaves.

2. Tidal Forces: The Moon's gravity would exert tidal forces on Earth, causing tides that would be much stronger than they currently are. This could result in more frequent and severe flooding, especially in coastal areas.

3. Earth's Rotation: The Moon's gravity would also affect Earth's rotation. As the Moon orbits around the Earth, it would cause the Earth to spin faster or slower depending on where it is relative to the Moon. This could lead to different seasons and potentially even changes in the length of day and night.

4. Changes in Ocean Circulation: The Moon's gravity would alter ocean currents and circulation patterns. This could lead to changes in sea levels, affecting coastal ecosystems and potentially leading to saltwater intrusion into freshwater sources.

5. Weather Patterns: The Moon's

DPO 後模型

If the Moon suddenly disappeared from our solar system, it would have profound and far-reaching effects on Earth and its inhabitants. Here's a detailed breakdown of what might happen:

1. Global Climate Change: The absence of the Moon would cause significant changes in Earth's climate. Without the gravitational pull of the Moon, the planet would experience less tidal forces, leading to more frequent and intense storms, hurricanes, and other weather phenomena.

2. Tidal Forces: Tides would become weaker or disappear entirely due to the lack of gravity. This could result in higher sea levels, which would lead to flooding and coastal erosion issues for many regions around the world.

3. Earth's Rotation: The rotation of the Earth would slow down significantly. This is because the Moon's gravity would no longer provide the necessary centripetal force to keep the Earth spinning at its current rate. As a result, the Earth would take much longer to complete one full rotation, potentially causing seasons to be shorter than they currently are.

4. Changes in Weather Patterns: The Moon's absence would also affect the Earth's weather patterns. The lack of the Moon's gravitational influence would mean that the Earth's atmosphere would not maintain its normal circulation, leading to unpredictable weather conditions such as droughts,

如何使用

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("steven0226/qwen2.5-0.5b-dpo-ultrafeedback", dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("steven0226/qwen2.5-0.5b-dpo-ultrafeedback")

由 tun0000/qwen-dpo-alignment 的 dpo_colab.ipynb 訓練產生。