toti-qwen-1.7b-v13 (GGUF Q4_K_M)
LoRA fine-tune Qwen3-1.7B untuk chatbot WhatsApp Toti Cakery (pemanggilan tool
bahasa Indonesia/Inggris, jawaban FAQ dari konteks RAG). Dilatih dari
dataset tool-calling Toti Cakery revisi v13
(finetune/data/ di github.com/kevinilhamramadhan/Chatbot-Cakery).
ollama pull hf.co/LasagnaS/toti-qwen-1.7b-v13-gguf:Q4_K_M
Evaluasi: model dasar vs fine-tuned
Diukur pada split test dataset yang sama, prompt persis seperti yang dikirim
chatbot di produksi, sampling temperature 0.7 / top_p 0.8, cap 192 token.
Rincian mentah: hasil_metrik_v13.json.
toti-qwen3-17b: sebelum vs sesudah fine-tuning (split test, 239 baris)
| Metrik | Arti | Sebelum | Sesudah | Selisih | Target | Status |
|---|---|---|---|---|---|---|
function_selection_acc |
baris tool: memilih tool yang benar | 0.188 | 0.868 | +0.680 | >= 0.85 | ✅ |
param_exact_acc |
baris tool: tool DAN argumen persis benar | 0.146 | 0.771 | +0.625 | >= 0.7 | ✅ |
irrelevance_acc |
baris non-tool: tidak memanggil tool | 0.958 | 0.926 | -0.032 | >= 0.85 | ✅ |
false_tool_rate |
baris non-tool: memanggil tool padahal tidak perlu | 0.042 | 0.074 | +0.032 | <= 0.1 | ✅ |
invalid_call_rate |
baris tool: panggilan tool rusak (JSON tak terbaca) | 0.049 | 0.000 | -0.049 | <= 0.05 | ✅ |
faq_grounded_acc |
FAQ: jawaban memuat fakta dari konteks (N1) | 0.867 | 0.667 | -0.200 | >= 0.7 | ❌ |
faq_jujur_acc |
FAQ: konteks tak memuat jawaban -> mengaku belum tahu (N1x) | 0.000 | 0.750 | +0.750 | >= 0.7 | ✅ |
wrong_language_rate |
balasan teks: meleset dari bahasa sesi pelanggan | 0.316 | 0.000 | -0.316 | <= 0.03 | ✅ |
Target tercapai: 7/8 · waktu training 236.6 menit · jawaban terpotong cap 192 token: sebelum 145, sesudah 0
Per tipe skenario
| Tipe | Skenario | n | Sebelum | Sesudah | Selisih |
|---|---|---|---|---|---|
| N1 | FAQ dari konteks | 15 | 0.87 | 0.67 | -0.20 |
| N1x | konteks tanpa jawaban | 4 | 0.00 | 0.75 | +0.75 |
| N2 | info belum ada | 3 | 1.00 | 0.67 | -0.33 |
| N3 | sapaan / terima kasih | 5 | 1.00 | 1.00 | +0.00 |
| N4 | di luar topik toko | 6 | 1.00 | 1.00 | +0.00 |
| N5 | tanya balik (ambigu) | 7 | 1.00 | 0.71 | -0.29 |
| N6 | jebakan (bukan aksi) | 8 | 1.00 | 0.88 | -0.12 |
| N7 | injeksi / rahasia / meta | 4 | 1.00 | 1.00 | +0.00 |
| N8 | sapaan pendek | 5 | 1.00 | 1.00 | +0.00 |
| N9 | minta orang / nego | 3 | 1.00 | 1.00 | +0.00 |
| N10 | bukan Owner minta laporan | 3 | 1.00 | 1.00 | +0.00 |
| N11 | ganti cara bayar | 4 | 0.25 | 1.00 | +0.75 |
| N12 | pesan pendek ambigu | 3 | 1.00 | 1.00 | +0.00 |
| N13 | bahasa percakapan | 3 | 1.00 | 1.00 | +0.00 |
| N14 | pengisi kontras (id/en) | 6 | 1.00 | 1.00 | +0.00 |
| N15 | N15 | 3 | 1.00 | 0.67 | -0.33 |
| N16 | N16 | 8 | 0.88 | 1.00 | +0.12 |
| N18 | N18 | 5 | 1.00 | 0.60 | -0.40 |
| T1 | menu | 7 | 0.29 | 1.00 | +0.71 |
| T2 | menu per kategori | 3 | 0.00 | 1.00 | +1.00 |
| T3 | detail produk | 12 | 0.08 | 0.75 | +0.67 |
| T4 | bandingkan produk | 4 | 0.00 | 1.00 | +1.00 |
| T5 | pesan 1 produk | 18 | 0.00 | 0.94 | +0.94 |
| T6 | pesan banyak produk | 4 | 0.00 | 0.75 | +0.75 |
| T7 | jawaban jumlah sesudah detail | 6 | 0.00 | 1.00 | +1.00 |
| T8 | status pesanan | 6 | 0.33 | 0.67 | +0.33 |
| T9 | batalkan pesanan | 3 | 0.67 | 1.00 | +0.33 |
| T10 | kue custom -> admin | 3 | 0.00 | 0.00 | +0.00 |
| T11 | laporan keuangan (Owner) | 3 | 0.33 | 1.00 | +0.67 |
| T12 | analitik (Owner) | 3 | 0.00 | 1.00 | +1.00 |
| T13 | klaim sudah bayar | 4 | 0.25 | 1.00 | +0.75 |
| T14 | keluhan | 4 | 0.00 | 0.00 | +0.00 |
| T15 | isi keranjang | 5 | 0.40 | 0.60 | +0.20 |
| T16 | kirim ulang cara bayar | 6 | 0.17 | 1.00 | +0.83 |
| T17 | T17 | 9 | 0.00 | 1.00 | +1.00 |
| T18 | T18 | 5 | 0.00 | 1.00 | +1.00 |
| T19 | T19 | 3 | 0.00 | 1.00 | +1.00 |
| T20 | T20 | 5 | 0.00 | 0.00 | +0.00 |
| T21 | T21 | 5 | 0.00 | 0.20 | +0.20 |
| T22 | T22 | 4 | 0.00 | 0.25 | +0.25 |
| T23 | T23 | 6 | 0.33 | 0.67 | +0.33 |
| T24 | T24 | 6 | 0.83 | 1.00 | +0.17 |
| T25 | T25 | 4 | 0.50 | 1.00 | +0.50 |
| T26 | T26 | 3 | 0.00 | 0.67 | +0.67 |
| T27 | T27 | 3 | 0.00 | 0.33 | +0.33 |