MetisRCA-E4B

Version 1.0.0. Fine-tune of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 published as ThakiCloud/MetisRCA-E4B.

Korean card: MODEL_CARD.ko.md

MetisRCA-E4B is a small incident-diagnosis model that reads a symptom, a list of runbook candidate causes and read-only observations, and answers with one JSON object: the root cause it can prove, the observation ids that decide it, or INSUFFICIENT_EVIDENCE with the next read-only tool to run. It is strong at picking the deciding evidence and at abstaining. On mechanisms it never saw in training it is below Qwen3.5-4B (Cloud-OpsBench like-for-like 0.783 vs 0.831) and it over-abstains on about 15% of determinable unseen-mechanism rows. An optional inference-time abstain policy (v4-calibrated, same weights, abstain_policy.py) lifts never-seen-mechanism top-1 to 0.824 (Cloud-OpsBench like-for-like 0.828, one row behind Qwen's 0.831) and cuts over-abstention to 9.4% with no rise in fabricated causes.

What it does

One call per diagnosis state. The system prompt fixes the contract; the user message carries the symptom, the candidate causes (cause_id: description) and the observations gathered so far, each with an id. The model replies with exactly one JSON object {"root_cause", "evidence_ids", "next_tool", "confidence_state"}. When the observations do not determine the cause it must answer INSUFFICIENT_EVIDENCE and name the next tool, so an agent loop can fetch more evidence and call it again. Labels come from benchmark ground truth and rules, not from a language model.

Intended use

• Evidence-selection and abstention layer inside an incident-diagnosis agent loop, with a human or a stronger model reviewing the verdict.
• Research on small-model abstention and evidence grounding in root-cause analysis.
• Starting point for your own fine-tune on a different runbook or telemetry format.

How to use

Prompt format

The model uses the base checkpoint's chat template with thinking disabled (enable_thinking=False). The rendered prompt is exactly:

<bos><|turn>system
You are an incident diagnosis assistant for a Kubernetes/GPU/DB AI platform. You get a symptom, the runbook candidate causes (cause_id: description) and read-only observations, each with an id. Reply with ONLY a JSON object {"root_cause": <one listed cause_id or "INSUFFICIENT_EVIDENCE">, "evidence_ids": [ids of the observations that decide it], "next_tool": <read-only tool to inspect next when unresolved, else null>, "confidence_state": "RESOLVED" | "UNRESOLVED"}. Decide only from the observations; if they do not determine the cause, answer INSUFFICIENT_EVIDENCE and name the next tool.<turn|>
<|turn>user
Symptom: Train Ticket: alert raised at t0 on the ts-ui-dashboard entry service. Since t0: p90 latency 0.5069 -> 0.6348.
Candidates:
- ts-train-service:mem: memory leak / memory stress in the ts-train-service container
- ts-auth-service:f2: code defect in ts-auth-service: missing parameters
- ts-train-service:disk: disk I/O stress on the ts-train-service container
- ts-order-service:f1: code defect in ts-order-service: incorrect parameter values
- ts-train-service:cpu: CPU hog in the ts-train-service container
Observations:
[e1] (query_resource_metrics)
service: ts-auth-service | window: 10 min before vs 10 min after alert
metric       |     before |      after | z
cpu          |     14.601 |     15.076 | z=0.3
mem          |  2.375e+08 |  2.395e+08 | z=0.2
diskio       |    52699.3 |    49751.0 | z=-0.2
socket       |     11.060 |     11.300 | z=0.3
[e2] (search_logs)
service: ts-auth-service | log lines before 3356 after 3312
metric       |     before |      after | z
error_lines  |          0 |          0 | z=0.0
[e3] (search_logs)
service: ts-order-service | log lines before 3434 after 3824
metric       |     before |      after | z
error_lines  |          0 |          0 | z=0.0<turn|>
<|turn>model

Output format: one compact JSON object: {"root_cause": <listed cause_id or "INSUFFICIENT_EVIDENCE">, "evidence_ids": [observation ids], "next_tool": <read-only tool or null>, "confidence_state": "RESOLVED"|"UNRESOLVED"}.

Call the model once per diagnosis state. When it answers INSUFFICIENT_EVIDENCE, run the named read-only tool, append the new observation to the user message and call again.

v4-calibrated (optional, transformers only): pass CalibratedAbstain from abstain_policy.py as a logits processor with greedy decoding. It forbids the abstain answer when the model prefers it by a logit margin below 1.5 at the first root_cause token, so the model names its best listed cause instead. inference_example.py runs both modes on example_incident.json. The policy changes no weights; GGUF runs give greedy v4 behavior.

Transformers + PEFT adapter

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "ThakiCloud/MetisRCA-E4B", subfolder="adapter").eval()

SYSTEM = "You are an incident diagnosis assistant for a Kubernetes/GPU/DB AI platform. You get a symptom, the runbook candidate causes (cause_id: description) and read-only observations, each with an id. Reply with ONLY a JSON object {\"root_cause\": <one listed cause_id or \"INSUFFICIENT_EVIDENCE\">, \"evidence_ids\": [ids of the observations that decide it], \"next_tool\": <read-only tool to inspect next when unresolved, else null>, \"confidence_state\": \"RESOLVED\" | \"UNRESOLVED\"}. Decide only from the observations; if they do not determine the cause, answer INSUFFICIENT_EVIDENCE and name the next tool."
USER = "Symptom: Train Ticket: alert raised at t0 on the ts-ui-dashboard entry service. Since t0: p90 latency 0.5069 -> 0.6348.\nCandidates:\n- ts-train-service:mem: memory leak / memory stress in the ts-train-service container\n- ts-auth-service:f2: code defect in ts-auth-service: missing parameters\n- ts-train-service:disk: disk I/O stress on the ts-train-service container\n- ts-order-service:f1: code defect in ts-order-service: incorrect parameter values\n- ts-train-service:cpu: CPU hog in the ts-train-service container\nObservations:\n[e1] (query_resource_metrics)\nservice: ts-auth-service | window: 10 min before vs 10 min after alert\nmetric       |     before |      after | z\ncpu          |     14.601 |     15.076 | z=0.3\nmem          |  2.375e+08 |  2.395e+08 | z=0.2\ndiskio       |    52699.3 |    49751.0 | z=-0.2\nsocket       |     11.060 |     11.300 | z=0.3\n[e2] (search_logs)\nservice: ts-auth-service | log lines before 3356 after 3312\nmetric       |     before |      after | z\nerror_lines  |          0 |          0 | z=0.0\n[e3] (search_logs)\nservice: ts-order-service | log lines before 3434 after 3824\nmetric       |     before |      after | z\nerror_lines  |          0 |          0 | z=0.0"
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": USER},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,
                               return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Merged BF16

The LoRA delta is merged into the base weights (BF16).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("ThakiCloud/MetisRCA-E4B")
model = AutoModelForCausalLM.from_pretrained("ThakiCloud/MetisRCA-E4B", dtype=torch.bfloat16, device_map="auto").eval()

SYSTEM = "You are an incident diagnosis assistant for a Kubernetes/GPU/DB AI platform. You get a symptom, the runbook candidate causes (cause_id: description) and read-only observations, each with an id. Reply with ONLY a JSON object {\"root_cause\": <one listed cause_id or \"INSUFFICIENT_EVIDENCE\">, \"evidence_ids\": [ids of the observations that decide it], \"next_tool\": <read-only tool to inspect next when unresolved, else null>, \"confidence_state\": \"RESOLVED\" | \"UNRESOLVED\"}. Decide only from the observations; if they do not determine the cause, answer INSUFFICIENT_EVIDENCE and name the next tool."
USER = "Symptom: Train Ticket: alert raised at t0 on the ts-ui-dashboard entry service. Since t0: p90 latency 0.5069 -> 0.6348.\nCandidates:\n- ts-train-service:mem: memory leak / memory stress in the ts-train-service container\n- ts-auth-service:f2: code defect in ts-auth-service: missing parameters\n- ts-train-service:disk: disk I/O stress on the ts-train-service container\n- ts-order-service:f1: code defect in ts-order-service: incorrect parameter values\n- ts-train-service:cpu: CPU hog in the ts-train-service container\nObservations:\n[e1] (query_resource_metrics)\nservice: ts-auth-service | window: 10 min before vs 10 min after alert\nmetric       |     before |      after | z\ncpu          |     14.601 |     15.076 | z=0.3\nmem          |  2.375e+08 |  2.395e+08 | z=0.2\ndiskio       |    52699.3 |    49751.0 | z=-0.2\nsocket       |     11.060 |     11.300 | z=0.3\n[e2] (search_logs)\nservice: ts-auth-service | log lines before 3356 after 3312\nmetric       |     before |      after | z\nerror_lines  |          0 |          0 | z=0.0\n[e3] (search_logs)\nservice: ts-order-service | log lines before 3434 after 3824\nmetric       |     before |      after | z\nerror_lines  |          0 |          0 | z=0.0"
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": USER},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,
                               return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

llama.cpp (GGUF)

llama-server -m MetisRCA-E4B-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080

Chat completions apply the template embedded in the GGUF; disable thinking per request:

curl http://localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{"messages": [{"role": "system", "content": "You are an incident diagnosis assistant for a Kubernetes/GPU/DB AI platform. You get a symptom, the runbook candidate causes (cause_id: description) and read-only observations, each with an id. Reply with ONLY a JSON object {\"root_cause\": <one listed cause_id or \"INSUFFICIENT_EVIDENCE\">, \"evidence_ids\": [ids of the observations that decide it], \"next_tool\": <read-only tool to inspect next when unresolved, else null>, \"confidence_state\": \"RESOLVED\" | \"UNRESOLVED\"}. Decide only from the observations; if they do not determine the cause, answer INSUFFICIENT_EVIDENCE and name the next tool."}, {"role": "user", "content": "Symptom: Train Ticket: alert raised at t0 on the ts-ui-dashboard entry service. Since t0: p90 latency 0.5069 -> 0.6348.\nCandidates:\n- ts-train-service:mem: memory leak / memory stress in the ts-train-service container\n- ts-auth-service:f2: code defect in ts-auth-service: missing parameters\n- ts-train-service:disk: disk I/O stress on the ts-train-service container\n- ts-order-service:f1: code defect in ts-order-service: incorrect parameter values\n- ts-train-service:cpu: CPU hog in the ts-train-service container\nObservations:\n[e1] (query_resource_metrics)\nservice: ts-auth-service | window: 10 min before vs 10 min after alert\nmetric       |     before |      after | z\ncpu          |     14.601 |     15.076 | z=0.3\nmem          |  2.375e+08 |  2.395e+08 | z=0.2\ndiskio       |    52699.3 |    49751.0 | z=-0.2\nsocket       |     11.060 |     11.300 | z=0.3\n[e2] (search_logs)\nservice: ts-auth-service | log lines before 3356 after 3312\nmetric       |     before |      after | z\nerror_lines  |          0 |          0 | z=0.0\n[e3] (search_logs)\nservice: ts-order-service | log lines before 3434 after 3824\nmetric       |     before |      after | z\nerror_lines  |          0 |          0 | z=0.0"}], "temperature": 0, "chat_template_kwargs": {"enable_thinking": false}}'

The reported GGUF numbers were measured by sending token ids rendered with the Hugging Face chat template, so the prompt is byte-identical to the BF16 run. If you call /completion with a raw string, check that the server does not add a second <bos>.

Evaluation

Greedy decoding on a held-out test split, same prompts for every arm.

Test split: held-out diagnosis states: a seen split (345 rows) whose fault types and mechanisms occur in training, and an unseen split (1566 rows) whose benchmark fault types or mechanisms do not. Evaluation feeds the gold observation chain for each state and scores one reply per state.

Run conditions:

• Bolt base, zero-shot: greedy decode, enable_thinking=False, transformers 5.19.0; 1x B200
• MetisRCA-E4B: greedy decode, enable_thinking=False, transformers 5.19.0; 1x B200
• Qwen3.5-4B, zero-shot: greedy decode, enable_thinking=False, transformers 5.19.0; 1x B200

Like-for-like on never-seen mechanisms (determinable rows)

Measured with rows whose normalized mechanism does not occur in training and whose observations determine the cause. Caveats: On Cloud-OpsBench MetisRCA is 4.8 points below Qwen3.5-4B and 1.9 points above the Bolt base. RCAEval rows are rule-solvable by construction and are not a headline (RCAEval counts are out of 33 rows). With the v4-calibrated abstain policy the same weights reach 298/360, 1 row (0.3 points) below Qwen3.5-4B.

Whole unseen split (1566 rows)

Measured with hardware B200, 1 GPU; all rows of the held-out unseen split (benchmark fault types or mechanisms absent from training); includes abstain rows. Caveats: The Qwen row comes from the ledger row that reports root_cause_top1, evidence_f1 and fabricated_cause_rate on the same 1566 rows; the other arms report the same quantities as unseen_all_top1, evidence_f1_unseen and fabricated_unseen.

v4-calibrated: inference-time abstain policy (same weights)

Measured with hardware B200, 1 GPU; whole unseen split (1566 rows); top-1, over-abstain and correct-abstain columns are on its never-seen-mechanism subset (783 rows); threshold t=1.5 chosen on the validation split only, applied once. Caveats: Policy: at the first root_cause token, forbid the abstain token when its logit margin over the best other token is below 1.5 (abstain_policy.py). Pre-registered gates: top-1 on never-seen mechanisms >= 0.814 PASS (0.824), over-abstain <= 5% FAIL (9.4%), correct abstain >= 70% PASS (73.6%), fabricated <= 7.3% PASS (0.26%). The first-token margin sign matched the actual abstain output on 100% of unseen rows. GGUF/llama.cpp runs reproduce the greedy v4 row, not this one. Added latency was not measured; the policy adds one comparison at one decode step and no second pass.

Merged weights and GGUF on the seen test split (345 rows)

Measured with hardware B200, 1 GPU; test_seen; GGUF runs feed the Hugging Face chat-template token ids to llama-server (llama.cpp 344be0a), greedy. Caveats: All variants keep top-1; quantization costs about one point of correct-abstain (2 of 180 abstain rows). Leak probe (130 prompts asking for internal identifiers): 0 hits on merged BF16, Q4_K_M and Q4_K_M-NV. Single-stream decode on B200 measured 202.8 tok/s for one Q4 variant; per-variant speed and memory were not retained.

Training

Method: LoRA fine-tune of the base model, loss on assistant turns only. Values below are copied from the training record.

Data

Training rows combine incidents from RCAEval (MIT license) and Cloud-OpsBench (MIT license) with internal synthetic playbook rows. Every row is a diagnosis state: the same incident is shown at several evidence levels (not yet enough evidence, near miss, sufficient), so the model learns when to stop and when to ask for another tool. Labels come from the benchmark ground truth and code rules. Train, validation and test splits are disjoint by incident group, and the unseen test split holds benchmark fault types or mechanisms that never occur in training.

The training data is internal and is not released. This repository contains no dataset files.

Split sizes (rows): train 4362, val 379, test_seen 345, test_unseen 1566.

Limitations

• On mechanisms that never occur in training, greedy MetisRCA-E4B is below Qwen3.5-4B (Cloud-OpsBench like-for-like on determinable rows 0.783 vs 0.831; 1.9 points above the Bolt base). With the v4-calibrated policy it reaches 0.828 on the same rows, still 1 row below Qwen. Do not pick it for open-world diagnosis of unfamiliar failure types.
• It over-abstains: on never-seen mechanisms the greedy model answers INSUFFICIENT_EVIDENCE on about 15% of rows that the observations determine (9.4% with the v4-calibrated policy, which still misses its pre-registered 5% gate). Treat an abstention as a request for more evidence, not as a verdict.
• The v4-calibrated numbers need the transformers logits processor in abstain_policy.py; GGUF files under llama.cpp behave like greedy v4.
• RCAEval rows are rule-solvable by construction, so RCAEval scores say little about diagnosis skill and are not a headline.
• The runbook candidates and observation formats come from two public benchmarks plus internal synthetic playbook rows; a different telemetry format needs re-evaluation.
• Do not let the model trigger remediation on its own. It reads observations; it does not verify that an action is safe.
• Behavior outside the prompt format above (other output schemas, thinking mode, very long observation lists) was not evaluated.

License and attribution

• Model materials: governed by the OpenMDW License Agreement, version 1.1 (LICENSE.OpenMDW-1.1, https://openmdw.ai/license/1-1/). This repository is a derivative of nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 and ships a copy of the agreement.
• Origin: the NVIDIA base model was post-trained on Google Gemma 4, whose use is governed by the Apache License, Version 2.0 (LICENSE.Apache-2.0-Gemma).
• NVIDIA's governing-terms notice is retained verbatim in NOTICE.md, together with the origin chain and a statement of ThakiCloud's modifications.
• The fine-tuned weights add ThakiCloud's LoRA modifications; they do not change the terms above.