customer-support-qwen2.5-0.5b

LoRA adapter for Qwen/Qwen2.5-0.5B-Instruct, supervised-fine-tuned on bitext/Bitext-customer-support-llm-chatbot-training-dataset for single-turn customer-support response generation.

Task

Given a customer request, generate the support response. Training objective: causal LM with response-only loss masking — the loss is computed on the assistant turn only, the system prompt and the customer message are masked with -100.

Training data

Preprocessing: NFKC normalisation, whitespace collapsing, missing/empty removal, exact (instruction, response) de-duplication. Casing, punctuation and word forms are preserved. Splitting is group-aware on the normalised customer message, so the same query cannot appear in two splits.

Prompt format

The tokenizer's native Qwen chat template is used (tokenizer.apply_chat_template):

<|im_start|>system
You are a professional customer support assistant. Answer clearly, politely and concisely.<|im_end|>
<|im_start|>user
{customer_message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

Training configuration

Results

Generation settings: max_new_tokens=128, greedy decoding (do_sample=False) for the table below; the sampled configuration (temperature=0.7, top_p=0.9) is reported in metrics.json.

Best validation loss: 0.6129 · validation perplexity: 1.841

Test loss and perplexity are teacher-forced, token-weighted cross-entropy over the response tokens of 2677 test examples, computed on identical tokenised sequences for both models. Generation metrics use 300 test examples for the adapter and 150 for the base model.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "thealper2/customer-support-qwen2.5-0.5b"

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [
    {"role": "system", "content": "You are a professional customer support assistant. Answer clearly, politely and concisely."},
    {"role": "user", "content": "I haven't received my refund yet. Can you check its status?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

out = model.generate(**inputs, max_new_tokens=128, do_sample=True,
                     temperature=0.7, top_p=0.9,
                     pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• 0.5B parameters: limited reasoning, no reliable factual grounding.
• The training data is synthetic and template-based; responses contain placeholders such as {{Order Number}} and the model reproduces that style, including inventing placeholders that were not in the request.
• Single-turn only — no dialogue history is modelled.
• Policies, phone numbers, URLs and hours in the output are generated text, not retrieved facts.
• English only.
• max_new_tokens=128 truncates responses longer than the budget; the reference responses have a median length of 108 tokens.