Qwen3.5-4B-Wazuh-Triage-LoRA

Mehdi Challakh, RAGraph.ai. Internal build v1.8, released 26 October 2026 under Apache-2.0 as ragraph-ai/Qwen3.5-4B-Wazuh-Triage-LoRA, with the paper Fine-Tuning a 4B Language Model for Site-Specific Wazuh Alert Triage (v2, 26 October 2026; arXiv preprint in preparation, the identifier will be added here).

One minute of a simulated night: every log line scored, the window risk re-assessed on each line, one flag at the first root login and none on the site's routine. The timeline and per-line scores are written for illustration (source); the measured results are in the Evaluation section below.

A Wazuh alert-triage classifier built on Qwen3.5-4B with three LoRA adapters. It reads the alerts one host raised in one 10-minute window and gives, in one forward pass and without generating text:

• a log-odds score that each alert line is part of malicious activity;
• a score for the window as a whole. The window score that worked best is the highest line score.

It is meant to rank and filter alerts for an analyst at a site whose own benign activity it was fine-tuned on. It is not a general-purpose detector. Its main strength is knowing which admin activity is routine at that site, which is also why it does not carry over to a new site unchanged (see Limitations).

What is in this repository

The lab's own logs, the training windows and the rendered text of any window are not part of the release.

How it works

The model is read out like a classifier rather than run as a chatbot ("label-token readout"):

1. Input. The window is rendered as text, in this order:

   • a header giving the host's role, whether host history is available, a line legend and a card for every rule that fired (level, description, groups, ATT&CK technique);
   • one line per distinct alert, each ending in ?.

2. Token sequence. The prompt uses the Qwen3.5 chat template with thinking turned off:

   <|im_start|>system\n{SYSTEM}<|im_end|>\n<|im_start|>user\n{header}\nAlerts:\n
   {line 1} ?\n{line 2} ?\n...{line n} ?\n
   Window verdict: A (contains malicious activity) or B (benign)?<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n
   

3. Scores. At each ? token and at the last position, the score is the log-odds of the label tokens: z = logsumexp(h·W[{A, " A"}]) − logsumexp(h·W[{B, " B"}]).

   • Only the four label-token rows of the output embedding are used. No full-vocabulary logits are computed.
   • Context lines (earlier alerts) get no marker and are not judged.

4. Training loss. Binary cross-entropy on every labelled line plus the window verdict; lines labelled −1 are left out.

System prompt:

You are a SOC alert-triage classifier. You read Wazuh alerts raised by one host in one time window and label them. Each alert line ends with '?'. At each '?', answer A if that alert is part of malicious activity or B if it is benign. After all alerts, answer A if the window contains any malicious alert, otherwise B.

Example of a rendered window (DEDALE client, identities aliased per window):

Host H: Windows workstation. Baseline: available; 'new' marks a rule not seen on H in the previous 7 days.
Window: 10 min, 128 alerts in 40 lines (+6 earlier lines for context, not to be judged).
Alert lines: time since previous line | rule id | 'new' ... | ×N identical alerts | event. Aliases: ...
Rules:
r92213 L15 Executable file dropped in folder commonly used by malware [sysmon_eid11_detections] (T1105 Ingress Tool Transfer)
...
Alerts:
+0s r92213 ×5 sysmon:11 img=c:\program files (x86)\mozilla firefox\firefox.exe user=DOM1\U1 file=c:\users\U1\... ?
+58s r91811 ps:4104 script=Invoke-WebRequest -Uri 'http://I1/....zip' -UseBasicParsing -OutFile 'C:\Users\U1\AppData\Roaming\... ?

Rendering (required: the model only works on this format)

The input format is part of the model. Raw Wazuh JSON will not work. The rendering code shipped in render/ does the following:

• Collapses identical alerts into one line marked ×N.
• Aliases hosts, users, domains and IPs per window (H, U1, DOM1, I1 / E1).
• Removes timestamps, GUIDs, hashes, process and record IDs.
• Lower-cases Windows paths.
• Drops fields that only some Sysmon versions log.
• Unescapes XML entities.
• Decodes -EncodedCommand PowerShell payloads into the command line.
• Flags novelty with new / new-proc / new-acct: a rule, process image or account this host has not shown in its history.

Each of these steps removes a shortcut that an earlier version learned (paper, Section 8). Feeding the model text rendered any other way gives undefined scores.

Usage sketch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE, REV = "Qwen/Qwen3.5-4B", "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"
tok = AutoTokenizer.from_pretrained(BASE, revision=REV)
enc = lambda s: tok.encode(s, add_special_tokens=False)
A = sorted({enc(" A")[0], enc("A")[0]}); B = sorted({enc(" B")[0], enc("B")[0]})
MARK = enc(" ?"); assert len(MARK) == 1

def encode(system, header, lines, scored, final_q):
    ids = enc(f"<|im_start|>system\n{system}<|im_end|>\n<|im_start|>user\n") + enc(header) + enc("\nAlerts:\n")
    pos = []
    for text, sc in zip(lines, scored):
        ids += enc(text)
        if sc:
            pos.append(len(ids))          # the ' ?' token: its next-token logits are the answer
        ids += MARK + enc("\n")
    ids += enc(f"{final_q}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n")
    return ids, pos

@torch.no_grad()
def line_logodds(model, ids, pos):
    base = model.get_base_model()
    h = base.model(input_ids=torch.tensor([ids], device=model.device)).last_hidden_state[0, pos]
    W = base.get_output_embeddings().weight
    return (torch.logsumexp((h @ W[A].T).float(), -1) - torch.logsumexp((h @ W[B].T).float(), -1)).cpu()

base = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16, device_map="cuda")
z = []
for seed_dir in ("adapter_s1234", "adapter_s2", "adapter_s3"):
    m = PeftModel.from_pretrained(base, seed_dir); m.eval()
    z.append(line_logodds(m, *encode(SYSTEM, header, lines, scored, FINAL_Q)))
    base = m.unload()                     # restore the plain base model before the next adapter
z = torch.stack(z).mean(0)                # ensemble = mean log-odds per line
window_score = z.max().item()             # flag the window if window_score > site threshold (log-odds)

Intended use

• For: ranking and filtering Wazuh alerts from Windows hosts (Security, Sysmon, PowerShell channels) at a site whose benign activity is in the training data. The model is fine-tuned again as the site's routine changes.
• Threshold: choose it on the site's own recent validation days. Do not use p > 0.5: the training mix has far more attacks than a real day, so 0.5 is not calibrated. Our rule is the cut with the best window F1 on the site's val windows. For the 3-seed ensemble it was p > 0.65 on the lab and p > 0.91 for DEDALE.
• Not for: autonomous response, blocking, or any decision without an analyst. It is also not for sites, operating systems or log sources it was not trained on (Linux coverage comes only from one web-server corpus), and not for use as evidence about a person.

Training data

14,186 training windows and 2,637 validation windows. All of them are Wazuh 4.14.9 alerts produced by one manager configuration from events whose labels are known, replayed from public logs or from our own lab. Each epoch draws 4,610 windows by a fixed quota per pool, so that DEDALE's 8,000 benign windows do not dominate; the lab pools, taken whole, are 44% of an epoch.

Labels:

• Lenient line labels are used for training:
  • per event or per line for the public corpora;
  • the runner's journal for the lab;
  • process lineage for the attack captures.
• Strict labels (a lab alert stays malicious only if its process descends from the runner's channel root) are used for every evaluation.
• Unknown lines (label −1) are shown to the model but not scored. Unlabelled background in an attack capture is never treated as benign.
• Augmentation: attack captures are shown with the generic role "Windows host"; 30% of training windows get a generic host role, no host history, 15% field dropout or time jitter, regardless of their label.
• Host-history dropout: in training, host history is removed from half of the windows that have it.

Held out of training:

• three attack scenarios: LSASS dump via comsvcs, Kerberoasting, scheduled-task persistence (public captures of related techniques are in training, so this tests transfer from public analogues);
• six benign admin-script families that ran only on w6 and w8;
• every test day (lab w2, w4, w6) and DEDALE weeks 3–4.

Evaluation

All numbers use windows (an attack window holds at least one strict attack line) and the 3-seed ensemble. Each model's threshold is chosen on its own validation windows, never on a test day.

Each result cell gives TP / FP / FN / TN, then F1 and window AP (threshold-free). DEDALE benign windows are weighted to the full two weeks.

¹ Hashed word 1–2-gram logistic regression (2²⁰ features, C = 4, class-balanced) on the same rendered lines and training windows, lenient labels. ² Same window text, prompted for per-line probabilities as JSON; default reasoning effort, about 3,000 reasoning tokens per window; a line it did not score counts as not flagged. ³ Same prompt as DeepSeek, thinking on, served by vLLM. 6% of windows got no usable answer and count as no alert. ⁴ The label-token readout applied to the untrained base model: it carries no information before training (w6 AP 0.07 equals the attack share).

At line level on w6 (137 strict attack lines): this model catches 79 with 95 false-alarm lines, DeepSeek 104 with 737, the text model 75 with 195.

Lab days (paired bootstrap over windows, 2,000 resamples, 95% intervals, w6):

• vs the text model: F1 +0.30 [+0.20, +0.41], 42 fewer false alarms [30, 55], recall −0.04 [−0.17, +0.08].
• vs DeepSeek: F1 +0.40 [+0.28, +0.51], 59 fewer false alarms [44, 75], recall +0.04 [−0.12, +0.20].
• At matched recall: to catch 25 of w6's 28 attack windows this model needs 5 false-alarm windows, the text model 40, DeepSeek 70.

Where the false alarms are (w6, 406 benign windows, split by the scripts that ran in each window):

DEDALE: the text model ranks windows better.

• The text model catches all 24 APT windows at 8 false-alarm windows over two weeks; this model needs 84.
• Neither model's validation threshold transfers well to DEDALE.
• Earlier versions of this model, trained with less lab data, ranked DEDALE better (window AP 0.93–0.94). The lab data cost some DEDALE ranking.

Seeds vary. Single-seed w6 F1 was 0.68 / 0.71 / 0.85. Seed 2 collapsed on DEDALE (window AP 0.28). Use the ensemble.

Limitations

• The gain is site knowledge. Most of the advantage over the baselines comes from knowing the site's routine admin jobs.
  • It raises no false alarm on the 112 w6 windows of admin scripts it trained on (the text model, trained on the same windows, raises 29).
  • It flags 10 of the 39 windows of six script families held out of training (DeepSeek 19, text model 23, an older version that had never seen them 15).
  • Every attack type held out of training is caught by every model, the zero-shot ones included, because these attacks have public look-alikes. Transfer to new attack types is therefore not measured.
• No test on an unseen site yet. Both test environments (the lab and DEDALE) have earlier days in training. On a new site, expect it to flag the site's own remote-admin plumbing until it is fine-tuned on that site's benign days.
• Test reuse. Lab day w6 guided development from v1.6 to v1.8, so some adaptive overfitting to w6 is likely.
• Alert-only. The model only sees what Wazuh rules fire on. In the CAM-LDS dataset, only 25% of 883 attack steps raised any Wazuh alert.
• Protocol attacks are seen only through their command lines: AS-REP roasting 3 of 5 windows. DC-side Kerberos and replication events are not labelled.
• Small, synthetic tests. The tests are one real APT (24 windows, 143 alert lines) and three lab days with 9 attack scenarios. The lab is a QEMU domain of three Windows Server 2019 hosts driven by scripts, not real users.
• Window verdict head. The answer at the end of the window is weaker than the highest line score. Use the highest line score.

Ethics and data

All logs are public datasets or come from our own isolated lab. No production data and no personal data were used. Lab identities are GOAD's fictional defaults, and every identity is aliased per window before the model sees it. The attack scenarios are well-known public techniques run by standard tools against our own lab; the release adds no offensive capability. For the zero-shot baseline, only lab windows and windows of the public DEDALE dataset were sent to the DeepSeek API.

Licences

• This release (adapters, rendering code, prompts, thresholds, per-window scores): Apache-2.0.
• Base model: Qwen3.5-4B, Apache-2.0.
• Public sources named here: DEDALE (CC-BY-4.0), Splunk attack_data (Apache-2.0), OTRF Security-Datasets (MIT), EVTX-to-MITRE-Attack (CC0), Sigma regression files (DRL-1.1, with author attribution). The Wazuh rule descriptions printed in each window come from the Wazuh ruleset (GPLv2).

Citation

@misc{challakh2026wazuhtriage,
  author = {Challakh, Mehdi},
  title  = {Fine-Tuning a 4B Language Model for Site-Specific Wazuh Alert Triage},
  year   = {2026},
  month  = oct,
  note   = {Version 2, arXiv preprint in preparation. Model: https://huggingface.co/ragraph-ai/Qwen3.5-4B-Wazuh-Triage-LoRA (build v1.8, three LoRA adapters on Qwen3.5-4B)}
}