Checkpoints from an end-to-end post-training run on Qwen/Qwen3-0.6B (~600M parameters) covering continued pre-training (CPT) → supervised fine-tuning (SFT) → DPO → reasoning-format warm-up → GRPO.
This is a checkpoint repository, not a single deployable model. Pick the
subfolder for the stage you want. The last stage (grpo_qwen06_merged) is the
"final" model; grpo_qwen06/llm_best_0 is the best GRPO adapter by eval reward.
Produced by NeelContractor/finetuning_recipes, based on the avbiswas/finetuning_recipes course repo. Built with Unsloth, TRL, and PEFT on Kaggle 2× Tesla T4.
┌──────────────► dpo_qwen06 (DPO, side branch)
│
Qwen3-0.6B ──► CPT ──► SFT ──┬──► reasoning_warmup ──► GRPO ──► final
│
└──► DPO
Two branches leave SFT: preference optimization (DPO) and the reasoning branch
(format warm-up, then GRPO). GRPO is trained from reasoning_warmup_qwen06_merged,
not from the DPO adapter.
| Path | Type | Stage | Base / notes |
|---|---|---|---|
models/cpt_qwen06/final + checkpoint-20…131 |
LoRA adapter | CPT on arXiv paper text | unsloth/Qwen3-0.6B-unsloth-bnb-4bit |
models/cpt_qwen06_merged |
Full model (fp) | merged CPT | Qwen3-0.6B |
models/sft_qwen06/final + checkpoint-100/150/198 |
LoRA adapter | Instruction SFT | cpt_qwen06_merged |
models/sft_qwen06_merged |
Full model (fp) | merged SFT | Qwen3-0.6B |
models/dpo_qwen06/final + checkpoint-50/77 |
LoRA adapter (+ ref/) |
DPO preference tuning | sft_qwen06_merged |
models/reasoning_warmup_qwen06/final + checkpoint-250/300/324 |
LoRA adapter | Reasoning-format SFT (think/answer) |
sft_qwen06_merged |
models/reasoning_warmup_qwen06_merged |
Full model (fp) | merged warm-up | Qwen3-0.6B |
models/grpo_qwen06/llm |
LoRA adapter | GRPO (latest) | reasoning_warmup_qwen06_merged |
models/grpo_qwen06/llm_best_0 |
LoRA adapter | GRPO (best eval reward) | reasoning_warmup_qwen06_merged |
models/grpo_qwen06_merged |
Full model (fp) | final merged model | Qwen3-0.6B |
models/grpo_qwen06/logs |
JSON/JSONL | GRPO train + eval logs | — |
Notes on naming:
final/ — last adapter saved for the stage.checkpoint-N/ — resumable checkpoints (adapter + optimizer/scheduler/RNG state).*_merged/ — LoRA merged into the base, saved as a full-precision
Qwen3ForCausalLM (safe to load directly with transformers).from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "NeelContractor/finetuning-checkpoints"
SUB = "models/grpo_qwen06_merged"
tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder=SUB)
model = AutoModelForCausalLM.from_pretrained(
REPO, subfolder=SUB, torch_dtype="auto", device_map="auto"
)
The adapter configs were written on Kaggle and reference local paths
(/kaggle/working/...), so load the matching merged base explicitly:
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "NeelContractor/finetuning-checkpoints"
base = AutoModelForCausalLM.from_pretrained(
REPO, subfolder="models/reasoning_warmup_qwen06_merged",
torch_dtype="auto", device_map="auto",
)
model = PeftModel.from_pretrained(
base, REPO, subfolder="models/grpo_qwen06/llm_best_0"
)
tokenizer = AutoTokenizer.from_pretrained(
REPO, subfolder="models/grpo_qwen06/llm_best_0"
)
The reasoning branch expects a system prompt that asks for a think block:
SYSTEM = (
"You are a helpful, respectful and honest assistant. You are an expert in "
"AI, deep learning, and machine learning research. Generate your reasoning "
"first inside <think> and </think> tags. After </think>, generate only "
"the requested final response. When JSON is requested, the content after "
"</think> must contain only that format, without Markdown fences."
)
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": "Explain LoRA in two sentences."},
]
text = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(
**inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9
)
print(tokenizer.decode(
output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
))
| Base model | Qwen/Qwen3-0.6B (CPT loaded via unsloth/Qwen3-0.6B-unsloth-bnb-4bit) |
| Method | LoRA (PEFT) with Unsloth kernels / TRL trainers |
| LoRA rank | r=32, lora_alpha=32 (CPT/SFT/DPO/warm-up), lora_alpha=64 (GRPO) |
| Target modules | q, k, v, o, gate, up, down projections |
| LoRA dropout | 0.0 |
| Hardware | Kaggle, 2× Tesla T4 (15 GB), DDP |
| Precision | fp16 training on T4; merged checkpoints saved as bf16 |
| Max sequence length | 1024 (SFT / reasoning / GRPO) |
| Reasoning warm-up | 10,000 sampled → 6,911 after the 1024-token filter; 3 epochs, 324 steps, ~46 min on 2×T4 |
| GRPO | 1 epoch, 8 train events (384 optimizer steps), ~50 min on 2×T4 |
| Stage | Data |
|---|---|
| CPT | arXiv paper text prepared by data_prep/cpt |
| SFT | paperbd/paper_instructions_300K-v1 (default config, ~20k rows) |
| DPO | Preference pairs produced/ranked by the preference-optimization scripts |
| Reasoning warm-up | reasoning config of paperbd/paper_instructions_300K-v1 |
| GRPO | Train slice of the same reasoning config, scored by paperbd/neuraltxt-reward-22M plus format/schema/length terms |
From models/grpo_qwen06/logs/eval_summary_5.json (64 held-out prompts):
| Metric | Mean |
|---|---|
total_reward |
2.355 |
neuraltxt_reward |
0.519 |
length_penalty_reward |
−0.078 |
num_tokens |
190.6 |
think_format_reward |
0.953 |
output_datatype_reward |
0.461 |
output_schema_reward |
0.461 |
Datatype and schema match rate: 93.8% each. Best inference score: 2.355.
An earlier run used a reasoning warm-up of only 3 optimizer steps, so the model
never emitted think... blocks and every format reward stayed at zero. The
current checkpoints come from the dedicated reasoning stage (10,000 rows,
3 epochs), which fixed it: 95.3% of eval responses contain a valid
think... block and 93.8% match the requested datatype/schema. Semantic
quality is still modest — this is a 0.6B research model, not a strong reasoner.
The orchestrator builds the data, runs each stage, and optionally pushes checkpoints to the Hub:
python cloud_train.py --install --num-gpus 2 \
--push-to-hub --hub-repo NeelContractor/finetuning-checkpoints
Run only the stages you need:
# Retrain the reasoning-format warm-up, then GRPO
python cloud_train.py --stages reasoning grpo --num-gpus 2
See the project README for stage-by-stage details, dataset prep, and evaluation scripts. The Kaggle runbook for the reasoning branch is REASON_TRAINING.md.
models/cpt_qwen06_merged/model.safetensors 1.2 GBmodels/grpo_qwen06_merged/model.safetensors 1.2 GBmodels/reasoning_warmup_qwen06_merged/model.safetensors 1.2 GBmodels/sft_qwen06_merged/model.safetensors 1.2 GBmodels/grpo_qwen06/llm/optimizer.pt 162 MBmodels/grpo_qwen06/llm_best_0/optimizer.pt 162 MBmodels/cpt_qwen06/checkpoint-100/adapter_model.safetensors 81 MBmodels/cpt_qwen06/checkpoint-120/adapter_model.safetensors 81 MBmodels/cpt_qwen06/checkpoint-131/adapter_model.safetensors 81 MBmodels/cpt_qwen06/checkpoint-20/adapter_model.safetensors 81 MBmodels/cpt_qwen06/checkpoint-40/adapter_model.safetensors 81 MBmodels/cpt_qwen06/checkpoint-60/adapter_model.safetensors 81 MBmodels/cpt_qwen06/checkpoint-80/adapter_model.safetensors 81 MBmodels/cpt_qwen06/final/adapter_model.safetensors 81 MBmodels/dpo_qwen06/checkpoint-50/adapter_model.safetensors 81 MBmodels/dpo_qwen06/checkpoint-50/ref/adapter_model.safetensors 81 MBmodels/dpo_qwen06/checkpoint-77/adapter_model.safetensors 81 MBmodels/dpo_qwen06/checkpoint-77/ref/adapter_model.safetensors 81 MBmodels/dpo_qwen06/final/adapter_model.safetensors 81 MBmodels/dpo_qwen06/final/ref/adapter_model.safetensors 81 MBmodels/grpo_qwen06/llm/adapter_model.safetensors 81 MBmodels/grpo_qwen06/llm_best_0/adapter_model.safetensors 81 MBmodels/reasoning_warmup_qwen06/checkpoint-250/adapter_model.safetensors 81 MBmodels/reasoning_warmup_qwen06/checkpoint-3/adapter_model.safetensors 81 MBmodels/reasoning_warmup_qwen06/checkpoint-300/adapter_model.safetensors 81 MBmodels/reasoning_warmup_qwen06/checkpoint-324/adapter_model.safetensors 81 MBmodels/reasoning_warmup_qwen06/final/adapter_model.safetensors 81 MBmodels/sft_qwen06/checkpoint-100/adapter_model.safetensors 81 MBmodels/sft_qwen06/checkpoint-150/adapter_model.safetensors 81 MBmodels/sft_qwen06/checkpoint-198/adapter_model.safetensors 81 MBmodels/sft_qwen06/final/adapter_model.safetensors 81 MBmodels/reasoning_warmup_qwen06/checkpoint-300/optimizer.pt 41 MBmodels/reasoning_warmup_qwen06/checkpoint-324/optimizer.pt 41 MBmodels/dpo_qwen06/checkpoint-50/optimizer.pt 41 MBmodels/dpo_qwen06/checkpoint-77/optimizer.pt 41 MBmodels/reasoning_warmup_qwen06/checkpoint-250/optimizer.pt 41 MBmodels/reasoning_warmup_qwen06/checkpoint-3/optimizer.pt 41 MBmodels/sft_qwen06/checkpoint-100/optimizer.pt 41 MBmodels/sft_qwen06/checkpoint-150/optimizer.pt 41 MBmodels/sft_qwen06/checkpoint-198/optimizer.pt 41 MBmodels/cpt_qwen06/checkpoint-100/optimizer.pt 41 MBmodels/cpt_qwen06/checkpoint-120/optimizer.pt 41 MBmodels/cpt_qwen06/checkpoint-131/optimizer.pt 41 MBmodels/cpt_qwen06/checkpoint-20/optimizer.pt 41 MBmodels/cpt_qwen06/checkpoint-40/optimizer.pt 41 MBmodels/cpt_qwen06/checkpoint-60/optimizer.pt 41 MBmodels/cpt_qwen06/checkpoint-80/optimizer.pt 41 MBmodels/cpt_qwen06/checkpoint-100/rng_state_0.pth 14.9 kBmodels/cpt_qwen06/checkpoint-100/rng_state_1.pth 14.9 kBmodels/cpt_qwen06/checkpoint-120/rng_state_0.pth 14.9 kBmodels/cpt_qwen06/checkpoint-120/rng_state_1.pth 14.9 kBmodels/cpt_qwen06/checkpoint-131/rng_state_0.pth 14.9 kBmodels/cpt_qwen06/checkpoint-131/rng_state_1.pth 14.9 kBmodels/cpt_qwen06/checkpoint-20/rng_state_0.pth 14.9 kBmodels/cpt_qwen06/checkpoint-20/rng_state_1.pth 14.9 kBmodels/cpt_qwen06/checkpoint-40/rng_state_0.pth 14.9 kBmodels/cpt_qwen06/checkpoint-40/rng_state_1.pth 14.9 kBmodels/cpt_qwen06/checkpoint-60/rng_state_0.pth 14.9 kBmodels/cpt_qwen06/checkpoint-60/rng_state_1.pth 14.9 kBmodels/cpt_qwen06/checkpoint-80/rng_state_0.pth 14.9 kBmodels/cpt_qwen06/checkpoint-80/rng_state_1.pth 14.9 kBmodels/dpo_qwen06/checkpoint-50/rng_state_0.pth 14.9 kBmodels/dpo_qwen06/checkpoint-50/rng_state_1.pth 14.9 kBmodels/dpo_qwen06/checkpoint-77/rng_state_0.pth 14.9 kBmodels/dpo_qwen06/checkpoint-77/rng_state_1.pth 14.9 kBmodels/reasoning_warmup_qwen06/checkpoint-250/rng_state_0.pth 14.9 kBmodels/reasoning_warmup_qwen06/checkpoint-250/rng_state_1.pth 14.9 kBmodels/reasoning_warmup_qwen06/checkpoint-3/rng_state_0.pth 14.9 kBmodels/reasoning_warmup_qwen06/checkpoint-3/rng_state_1.pth 14.9 kBmodels/reasoning_warmup_qwen06/checkpoint-300/rng_state_0.pth 14.9 kBmodels/reasoning_warmup_qwen06/checkpoint-300/rng_state_1.pth 14.9 kBmodels/reasoning_warmup_qwen06/checkpoint-324/rng_state_0.pth 14.9 kBmodels/reasoning_warmup_qwen06/checkpoint-324/rng_state_1.pth 14.9 kBmodels/sft_qwen06/checkpoint-100/rng_state_0.pth 14.9 kBmodels/sft_qwen06/checkpoint-100/rng_state_1.pth 14.9 kBmodels/sft_qwen06/checkpoint-150/rng_state_0.pth 14.9 kBmodels/sft_qwen06/checkpoint-150/rng_state_1.pth 14.9 kBmodels/sft_qwen06/checkpoint-198/rng_state_0.pth 14.9 kBmodels/sft_qwen06/checkpoint-198/rng_state_1.pth 14.9 kBmodels/cpt_qwen06/checkpoint-100/training_args.bin 6.7 kBmodels/cpt_qwen06/checkpoint-120/training_args.bin 6.7 kBmodels/cpt_qwen06/checkpoint-131/training_args.bin 6.7 kBmodels/cpt_qwen06/checkpoint-20/training_args.bin 6.7 kBmodels/cpt_qwen06/checkpoint-40/training_args.bin 6.7 kBmodels/cpt_qwen06/checkpoint-60/training_args.bin 6.7 kBmodels/cpt_qwen06/checkpoint-80/training_args.bin 6.7 kBmodels/dpo_qwen06/checkpoint-50/training_args.bin 6.7 kBmodels/dpo_qwen06/checkpoint-77/training_args.bin 6.7 kBmodels/reasoning_warmup_qwen06/checkpoint-250/training_args.bin 6.6 kBmodels/reasoning_warmup_qwen06/checkpoint-300/training_args.bin 6.6 kBmodels/reasoning_warmup_qwen06/checkpoint-324/training_args.bin 6.6 kBmodels/reasoning_warmup_qwen06/checkpoint-3/training_args.bin 6.5 kBmodels/sft_qwen06/checkpoint-100/training_args.bin 6.5 kBmodels/sft_qwen06/checkpoint-150/training_args.bin 6.5 kBmodels/sft_qwen06/checkpoint-198/training_args.bin 6.5 kBmodels/dpo_qwen06/checkpoint-50/scheduler.pt 1.5 kBmodels/dpo_qwen06/checkpoint-77/scheduler.pt 1.5 kBmodels/reasoning_warmup_qwen06/checkpoint-250/scheduler.pt 1.5 kBmodels/reasoning_warmup_qwen06/checkpoint-3/scheduler.pt 1.5 kBmodels/reasoning_warmup_qwen06/checkpoint-300/scheduler.pt 1.5 kBmodels/reasoning_warmup_qwen06/checkpoint-324/scheduler.pt 1.5 kBmodels/sft_qwen06/checkpoint-100/scheduler.pt 1.5 kBmodels/sft_qwen06/checkpoint-150/scheduler.pt 1.5 kBmodels/sft_qwen06/checkpoint-198/scheduler.pt 1.5 kBmodels/cpt_qwen06/checkpoint-100/scheduler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-120/scheduler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-131/scheduler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-20/scheduler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-40/scheduler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-60/scheduler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-80/scheduler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-100/scaler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-120/scaler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-131/scaler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-20/scaler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-40/scaler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-60/scaler.pt 1.4 kBmodels/cpt_qwen06/checkpoint-80/scaler.pt 1.4 kBmodels/dpo_qwen06/checkpoint-50/scaler.pt 1.4 kBmodels/dpo_qwen06/checkpoint-77/scaler.pt 1.4 kBmodels/reasoning_warmup_qwen06/checkpoint-250/scaler.pt 1.4 kBmodels/reasoning_warmup_qwen06/checkpoint-3/scaler.pt 1.4 kBmodels/reasoning_warmup_qwen06/checkpoint-300/scaler.pt 1.4 kBmodels/reasoning_warmup_qwen06/checkpoint-324/scaler.pt 1.4 kBmodels/sft_qwen06/checkpoint-100/scaler.pt 1.4 kBmodels/sft_qwen06/checkpoint-150/scaler.pt 1.4 kBmodels/sft_qwen06/checkpoint-198/scaler.pt 1.4 kB有文件?提交种子。
提交种子Qwen/Qwen3-0.6B