⚡ Spark-Servitor

Experimental rule-assisted code diff gate with an unvalidated 1.7B neural head & OASIS SARIF v2.1.0 output

---

[!WARNING] Status: experimental. Do not use as a security gate.

• On an independent held-out benchmark (PrimeVul paired CVE fixes, 433 pairs) the shipped pipeline scores 50.0% accuracy with 1.8% recall on vulnerability-introducing diffs. That is chance level.
• Earlier versions of this card reported "93.5% Real-World PR Accuracy" and "100% Critical Vulnerability Recall". Those numbers were withdrawn: 93.5% is the training-set accuracy of the last training epoch (see What went wrong), and no run produced the 100% recall figure.
• Most verdicts produced by the --weights / native path come from string-matching rules and a taint extractor, not from neural inference.
• Full evaluation details are in Evaluation.

---

Overview

Spark-Servitor is a code diff screening tool. It canonicalizes a unified diff, runs it through rule-based and taint-flow checks, and (optionally) a 1.7B-parameter neural verdict/risk/pathology head, then emits a verdict (APPROVE / QUARANTINE / REJECT), a risk score, pathology labels and an OASIS SARIF v2.1.0 report with exploit flows.

It is a sibling of wxsys/qwen-servitor.

What works today:

• Diff canonicalization, multi-file parsing, SARIF v2.1.0 emission, CLI / daemon / FFI plumbing. The repo test suite passes (149 passed, locally re-run).
• A rule/taint layer that catches classic web/shell patterns (e.g. os.system(user_input), string-concatenated SQL, eval on request data, double-lock deadlocks) and passes the safe forms of those patterns.

What does not work yet:

• Detection of vulnerabilities outside those patterns (memory safety, auth/logic flaws, IDOR, hardcoded secrets, verify=False, XSS sinks, …).
• A neural head that adds measurable value over the rules (see Evaluation).

---

Architecture (as implemented)

            [ unified diff ]
                   │
                   ▼
        DiffCanonicalizer (parse, tokenize)
                   │
      ┌────────────┼──────────────────────────┐
      ▼            ▼                          ▼
 Rule / taint   Native C++ engine        Neural cortex (PyTorch)
 extractor      (string-match heuristics)  SparkServitorModel
 (Python)       used by --weights /        28-layer SWA/full-attn backbone
                daemon / FFI path          + DiffCAMQP cortex head
      │            │                          │ (not used by the native path)
      └────────────┴──────────┬───────────────┘
                              ▼
                 Hierarchical veto + (QUARANTINE) Z3 SMT gate
                              ▼
               verdict · risk · pathologies · SARIF

Important implementation facts:

• The native C++ engine (src/spark_simd_engine.cpp) decides verdicts with substring checks such as system(, shlex.quote, Lock()/Unlock(). Weights are loaded into memory, but the verdict path does not run a forward pass over them.
• In spark_servitor/engine.py, an unmitigated taint flow forces REJECT with risk ≥ 0.95, overriding the neural output.
• The CLI defaults to --mock-mode (rule-based emulation) unless --weights is given.
• The PyTorch SparkServitorModel neural path exists and can be run by dequantizing a .servitor file; it was evaluated separately (below).

---

Specifications

The previously listed figures "4.08 MB resident memory" and "0.044 ms per diff" describe the C++ rule path only, were not independently re-measured, and are not neural-inference numbers. Loading the weights alone takes ~1.3–1.6 GB on disk.

---

Model Files

The repository does not contain model.safetensors or cortex_head.pt. Older docs that mention them are wrong.

---

Evaluation

All numbers below were produced on 2026-10-16 on a CPU-only machine with data not used for training.

1. PrimeVul paired test (real CVEs, C/C++) — shipped pipeline

Source: colin/PrimeVul, paired config, test split. 433 usable (vulnerable, fixed) function pairs. For each pair two diffs were built with difflib:

• fixed → vulnerable ("a PR that introduces the bug", label 1)
• vulnerable → fixed ("a PR that fixes the bug", label 0)

97.5% of pairs were approved on both sides. The rules do not target C/C++ memory-safety bugs. PrimeVul is a hard benchmark; this result does not claim the task is easy, only that this tool does not solve it.

2. Hand-written multi-language set (38 diffs) — shipped pipeline

8 obvious vulnerabilities, 10 subtle vulnerabilities, 10 "looks dangerous but safe" cases, 10 clean diffs (Python, Go, JS, C). Labels written by the evaluator.

The final pipeline equals the taint rules alone (0.68). Missed examples: strcpy overflow, hardcoded AWS key, innerHTML = location.hash, or True auth bypass, IDOR, verify=False, path traversal on upload. Wrongly rejected: ast.literal_eval, a realpath-checked file open.

3. Neural head run directly (PyTorch, dequantized INT4 weights, fp32, CPU)

[!NOTE] Tokenizer caveat. When these .servitor weights were trained, DiffCanonicalizer.tokenize_canonical_text mapped words to vocabulary ids using Python's process-salted hash() instead of the Spark-X2.5 tokenizer. DiffCanonicalizer now uses deterministic zlib.crc32 across processes, but the existing .servitor checkpoints remain mismatched with any inference token mapping until retrained in Phase 3.

As a gate the neural head flags almost everything. The 80% pairwise figure is not evidence of skill: a trivial "more deleted lines ⇒ riskier" rule reaches ~73% on the same pairs, and 50 pairs gives a wide confidence interval. Verdict probabilities for a plain strcpy overflow were near-uniform (0.37 / 0.36 / 0.28).

Limits of this evaluation

• Small samples (38 hand-written diffs, 50 neural pairs, tokens truncated to 512). Treat results as indicative, not precise.
• PrimeVul is C/C++ only; Python/JS/Go were covered only by the 38 hand-written diffs.
• The hand-written labels were created by the evaluator, not an independent annotator.
• Latency and memory were not re-measured for the neural path.

---

Known issues & what was wrong with earlier claims

---

Quickstart (what actually exists)

CLI

# Rule-based emulation (default, no weights)
git diff HEAD~1..HEAD | spark-servitor --format tree

# With the packed weights file (still rule/taint driven, see warnings above)
spark-servitor --file patch.diff --weights models/spark-servitor-awq_int4.servitor --format json

Resident daemon

spark-servitord --socket /run/user/1000/spark-servitor.sock --weights models/spark-servitor-awq_int4.servitor --daemon
spark-servitor --socket /run/user/1000/spark-servitor.sock --file patch.diff

Python

from spark_servitor.engine import SparkServitorEngine

engine = SparkServitorEngine(mock_mode=False, weights_path="models/spark-servitor-awq_int4.servitor")
verdict = engine.evaluate_diff(diff_text)

print(verdict.status, verdict.risk, verdict.pathology)

---

Intended use

• Research and development of diff-screening pipelines, SARIF plumbing, and as a lightweight rule-based pre-filter for a small set of web/shell patterns.
• Not suitable for gating merges, replacing SAST tools, or any decision where a missed vulnerability matters.

Roadmap

A plan to rebuild the neural component properly (pretrained backbone, real training data, held-out evaluation, honest reporting) is tracked in the project's improvement plan.

---

Related Repositories

• wxsys/qwen-servitor: Sister model based on Qwen3.5-0.8B Linear Attention (DeltaNet), specialized for sub-15ms local pre-commit hooks.