A task-specific LoRA adapter fine-tuned by ShwethaJ to turn English support tickets into structured issue categories, affected environments, reported impact summaries, and support-team recommendations.
This repository contains adapter weights, not a complete standalone model. Load them with the original Qwen/Qwen3-0.6B weights at the revision below. Publishing this repository does not create a hosted inference service.
Final evaluation used 66 held-out, AI-authored fictional tickets, 11 per issue category. The selected adapter was frozen before final test evaluation. Both models used the same prompt, base revision, greedy prediction settings, test inputs, and scoring code.
| Metric | Base model | Fine-tuned | Change |
|---|---|---|---|
| Schema-valid structured answers | 53.0% | 98.5% | +45.5 percentage points |
| Category macro-F1 | 0.471 | 0.881 | +0.411 F1 |
| Environment accuracy | 34.8% | 90.9% | +56.1 percentage points |
| Team accuracy | 18.2% | 90.9% | +72.7 percentage points |
Correct issue labels increased from 25/66 to 58/66. Responses failing the required output schema fell from 31/66 to 1/66. Rates and changes are independently rounded from full-precision results.
Scoring: invalid-schema responses count as incorrect for every task metric. Macro-F1 averages F1 equally across six categories, not percent correct. Team is deterministically derived from category and is not an independent reasoning capability. Summary factuality/completeness are not automatically measured by these scores.
Machine-readable comparison · Base-model metrics · Fine-tuned metrics
Input: one short English support ticket. Output: a JSON object with exactly four fields:
{
"category": "authentication",
"environment": "production",
"customer_reported_impact": "The customer cannot access the production dashboard.",
"suggested_team": "developer_support"
}
The JSON above illustrates the output contract; it is not a saved inference result.
authentication, billing, rate_limit, integration, bug, or other.production, staging, development, or unknown when not established.null when no impact is stated.billing_support for billing, general_support for other, and developer_support for remaining categories.Mixed independent requests with no dominant issue are labeled other. Tickets are treated as data rather than instructions. Predictions should be reviewed before business actions; the model does not resolve issues or execute changes.
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen3-0.6B |
| Exact base version | c1899de289a04d12100db370d81485cdf75e47ca |
| Data | 128 training / 34 validation / 66 test tickets |
| Provenance | AI-authored fictional examples; agent review and user acceptance for a pilot |
| Method | LoRA, rank 16, alpha 32, dropout 0.05 |
| Target layers | q_proj, k_proj, v_proj, o_proj |
| Trainable parameters | 4,587,520 (about 0.76% of model + adapter parameters) |
| Epochs completed | 3 |
| Selected version | Epoch 2, checkpoint-32 |
| Selection rule | Lowest validation answer-token loss: 0.256318 |
| Learning rate | 0.0002 |
| Microbatch / accumulation | 1 / 8 |
| Maximum sequence length | 512; overlength examples rejected rather than truncated |
| Loss | Assistant answer and end-of-turn token only; prompt/ticket labels masked |
| Compute | Google Colab Tesla T4; no owned GPU |
| Precision | Float32 parameters with FP16 mixed-precision training |
Validation guided checkpoint selection. Test tickets were not used for gradient updates or selecting the checkpoint. This small pilot does not demonstrate generalization to real customer traffic.
Use a Python environment with PyTorch installed for your hardware, then install the library versions used in the experiment:
pip install transformers==4.57.1 peft==0.17.1 accelerate==1.10.1
The original Colab run used PyTorch 2.14.1+cu130. Compatibility and speed on other machines should be checked. CPU inference is possible in the code below but is not benchmarked here.
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
from peft import PeftModel
ADAPTER = "ShwethaJ/qwen3-0.6b-support-ticket-triage-lora"
BASE = "Qwen/Qwen3-0.6B"
BASE_VERSION = "c1899de289a04d12100db370d81485cdf75e47ca"
# No authentication is normally needed for public, ungated downloads.
prompt_path = hf_hub_download(ADAPTER, "system_prompt.txt")
with open(prompt_path, encoding="utf-8") as handle:
system_prompt = handle.read().strip()
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (torch.bfloat16 if device == "cuda" and torch.cuda.is_bf16_supported()
else torch.float16 if device == "cuda" else torch.float32)
tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForCausalLM.from_pretrained(
BASE, revision=BASE_VERSION, torch_dtype=dtype, attn_implementation="eager"
).to(device)
model = PeftModel.from_pretrained(base, ADAPTER, is_trainable=False).eval()
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": "Our production dashboard rejects sign-in with a 401 error."},
]
inputs = tokenizer.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True,
enable_thinking=False, return_dict=True, return_tensors="pt",
).to(device)
config = GenerationConfig(
do_sample=False, num_beams=1, max_new_tokens=256,
eos_token_id=model.generation_config.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
use_cache=True,
)
with torch.inference_mode():
output = model.generate(**inputs, generation_config=config)
answer_ids = output[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(answer_ids, skip_special_tokens=True))
Use the supplied system prompt and non-thinking chat template for comparability. Validate generated JSON in your application. Greedy decoding is the chosen experimental baseline setting, not Qwen's default recommended sampling recipe. The example preserves raw output rather than silently repairing it.
For reproducible deployment, also pin this adapter repository to a full commit ID after publishing. The base revision is already fixed above. Training identity records configuration and data fingerprints.
Based on Qwen3-0.6B, released by the Qwen team under Apache 2.0. Adapter published under Apache 2.0. Respect the base model license and preserve its attribution when redistributing.
Qwen/Qwen3-0.6B