Last version was a mess, it hallucinate code, and can't even tell what's the diff of remote events or type of scripts so this might be the final attempt of fine-tuning with this model. So i fixed the dataset itself, i removed all of the remaining broken refs, fixed the dataset with ChatGPT. Finally my Colab runtime is worth it. (spoiler: next version will add basic roblox knowledge and will use 7b (so please expect smarter and maybe much better models))
Evaluated via True Performance Network (TPN Bench) using lm_eval (vLLM backend, plain prompts, commit 196f61e):
| Benchmark | Metric | Score | Stderr | Samples |
|---|---|---|---|---|
| ARC-Challenge | acc_norm |
47.95% | ±1.46% | 1,172 |
| ARC-Challenge | acc |
44.54% | ±1.45% | 1,172 |
This qwen2 model was trained 2x faster with Unsloth and Huggingface's TRL library.
model-00001-of-00002.safetensors 5.0 GBmodel-00002-of-00002.safetensors 1.2 GB