Qwen2.5-0.5B-Instruct-opensec-triage-lora

LoRA adapter fine-tune of Qwen/Qwen2.5-0.5B-Instruct on tegridydev/opensec-triage (research config) that maps a security alert with contextual evidence to a four-field structured triage decision.

Output format

Category: <endpoint|cloud_saas|identity_access|network|application_data|cloud_native>
Disposition: <malicious|suspicious|benign|expected_admin|authorised_testing|misconfiguration|unknown>
Severity: <high|medium|low|info>
Action: <escalate_incident_response|investigate_with_owner|collect_missing_evidence|fix_configuration|verify_change_record|verify_assessment_scope|document_and_close>

Prompt

System prompt:

You are a security operations triage assistant. Analyze the provided security alert and its evidence and produce a concise triage decision. Base the decision only on the evidence in the alert. Do not invent facts that are not present in the alert and do not add explanations.

User message template (Qwen2.5 chat template):

Alert:
{alert}

Respond exactly in this format:
Category: <endpoint|cloud_saas|identity_access|network|application_data|cloud_native>
Disposition: <malicious|suspicious|benign|expected_admin|authorised_testing|misconfiguration|unknown>
Severity: <high|medium|low|info>
Action: <escalate_incident_response|investigate_with_owner|collect_missing_evidence|fix_configuration|verify_change_record|verify_assessment_scope|document_and_close>

Training data

Training procedure

Evaluation

Greedy decoding (do_sample=False, max_new_tokens=48). Outputs are parsed per field (case/whitespace/punctuation normalised); a response is valid only if all four fields appear exactly once. Invalid fields count as errors.

Reference (non-LLM) baselines on the same test split: majority class disposition accuracy 0.1424; TF-IDF + logistic regression disposition accuracy 1.0 (OOD model_trainer/test: 0.8143), category accuracy 1.0.

• joint_triage_accuracy: all scored fields correct.
• high_severity_undertriage_rate: share of gold high examples predicted below high or invalid.
• internal_consistency_rate: predicted severity and action agree with the predicted disposition's mapping.
• Macro F1 is reported per field; per-class scores are in evaluation_results.json.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "thealper2/Qwen2.5-0.5B-Instruct-opensec-triage-lora").merge_and_unload()
tokenizer = AutoTokenizer.from_pretrained("thealper2/Qwen2.5-0.5B-Instruct-opensec-triage-lora")

import json
from huggingface_hub import hf_hub_download

schema = json.load(open(hf_hub_download("thealper2/Qwen2.5-0.5B-Instruct-opensec-triage-lora", "triage_schema.json")))

alert = "Identity audit on ASSET-... recorded that ... <evidence sentences>"
messages = [
    {"role": "system", "content": schema["system_prompt"]},
    {"role": "user", "content": f"Alert:\n{alert}\n\n{schema['response_format']}"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, do_sample=False, max_new_tokens=48)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

triage_schema.json (in this repository) contains the exact system prompt, response format, label vocabularies and mappings used in training. The prompt must match it exactly.

Limitations

• Training and test data are synthetic and strongly templated; a TF-IDF baseline reaches near-perfect accuracy on the in-distribution test split, so in-distribution scores mainly measure template fit. The OOD column is the more informative generalisation estimate.
• Severity is a fixed projection of disposition; the dataset itself does not assign severity.
• Decisions depend on the evidence sentences in the input (approvals, baselines, correlation). Bare log lines without such context are outside the training distribution.
• English only. Not evaluated on real SOC telemetry. Intended as a triage assistant, not an autonomous response system.