CodeAstra-500M: Laptop-Scale Vulnerability Detection 🔍🛡️

Model Description

CodeAstra-500M is the small sibling of CodeAstra-7B — a 500M-parameter sparse Mixture-of-Experts model fine-tuned for security vulnerability detection in source code. Where CodeAstra-7B is built on Mistral-7B, CodeAstra-500M is built on HobbyLM-Chat, a MoE language model trained from scratch on a hobby budget — so the whole thing runs on a laptop CPU.

It answers with a single structured JSON verdict: is this vulnerable, which CWE, which OWASP category.

Key Features

• 🪶 Tiny: 500M total parameters, only ~150M active per token thanks to top-6-of-36 expert routing.
• 📐 Structured output: emits parseable JSON on 99.98% of inputs — no regex-scraping prose.
• 🎚️ Tunable: returns a calibrated probability, so you pick the precision/recall trade-off at inference time instead of retraining.
• 🌐 Multi-language: C, C++, Python, Java, JavaScript, PHP, Go, Ruby, Swift, Kotlin, C#, Fortran — though the training mix is heavily C-weighted (see Limitations).
• 🧪 Honestly evaluated: scored on the standard split and on vulnerable/patched function pairs, which is the harder and more meaningful test.
• 🥊 Benchmarked against a frontier model: matches DeepSeek v4 Flash on genuine flaw discrimination while being ~10,000× faster per snippet.
• 💻 Runs locally: shares the HobbyLM architecture, so it loads in the from-scratch Rust CPU engine (hobby-rs) with no Python at runtime.

Performance 📊

Evaluated on the held-out test split (17,542 snippets, 1,887 vulnerable) of ayshajavd/code-security-vulnerability-dataset. The base model is un-finetuned HobbyLM-Chat under identical prompts.

⚠️ A note on accuracy. This dataset is 89% non-vulnerable, so a model that answers "safe" every time scores 90.5% accuracy — which is exactly what the un-finetuned base model does, at 0% recall. That is why this card leads with F1, AUC and recall rather than accuracy. Treat any headline accuracy figure on this dataset with suspicion, including for other models.

Paired evaluation — the number that actually matters 🎯

Standard splits of CVE-derived vulnerability datasets are confounded: the vulnerable functions come from big C projects (Linux kernel, Chromium, PHP, ffmpeg) while the "safe" ones are often unrelated code. A model can score very well by recognising code style rather than code flaws.

So CodeAstra-500M is also evaluated PrimeVul-style, on 894 pairs of a vulnerable function and its own patched version — near-identical code differing only by the security fix.

The released model was produced by feeding the ~7,000 patched functions back in as safe training examples — minimal pairs that differ only by the fix. This quadrupled paired accuracy and lifted specificity on patched code from 13.65% to 63.42%.

Read this honestly: within-pair ranking accuracy moved only 66.22% → 69.46%, meaning most of the improvement is better calibration rather than deeper understanding. 27.40% is the realistic estimate of true detection ability — not the 96.97% AUC. Very few vulnerability models publish this number; it is here because it is the one that predicts real-world behaviour.

Comparison against a frontier model 🥊

Benchmarked against DeepSeek v4 Flash (via OpenRouter, reasoning enabled) on the same held-out rows, through the same prompts, the same output parser and the same metrics. DeepSeek answers zero-shot; CodeAstra is fine-tuned on this corpus.

On the standard held-out split — 1,000 rows, 93 vulnerable

DeepSeek flagged 308 of 1,000 snippets as vulnerable when only 93 were — a 3.3× over-prediction, since a general model has no way to know this corpus is 89% safe.

⚠️ Do not read this as "CodeAstra is 3× better at security." Most of that gap is corpus familiarity — the base rate and the annotator's CWE conventions — not better security reasoning. The per-CWE breakdown makes it plain: DeepSeek gets CWE-89 (SQL injection) 100% detected and 80% exact, but 0% exact on CWE-119, a catch-all bucket whose boundaries you can only learn from the training set.

On the paired set — 894 pairs, the unconfounded test

On this harder, style-controlled test the two models are closely matched — CodeAstra is marginally ahead, holds higher specificity on patched code, and is 3× less likely to invert a pair.

✅ The defensible claim: CodeAstra-500M matches a frontier reasoning model on genuine flaw discrimination, at ~10,000× lower latency, no per-call cost, and fully local.

Distinguishing a vulnerable function from its patched twin is a hard problem at the single-function level for models of any size — see Limitations.

Intended Use

CodeAstra-500M is for developers, security researchers and code auditors who want a fast first-pass triage filter that runs locally — in a pre-commit hook, a CI step, or an editor plugin — without sending source code to an API. It is a filter that decides what a human looks at, not an oracle.

Threshold — please read 🎚️

The model returns a probability. The default 0.5 is not the balanced operating point.

At the naive 0.5 you get 88.79% precision but only 48.70% recall — it will quietly miss half the vulnerabilities. Set the threshold deliberately.

Training 🏋️‍♂️

Fine-tuned from HobbyLM-Chat on 8×H100 GPUs via Modal, using the full dataset — all 140,335 training rows, nothing subsampled, with over-long code head+tail truncated rather than dropped.

Architecture is unchanged from HobbyLM: 768 hidden / 16 layers, 36 experts with top-6 routing plus one shared expert, GQA attention with per-head QK-norm, RoPE, GPT-2 byte-level BPE, 2048-token context.

Usage 💻

The model uses the HobbyLM MoE architecture, so it needs the hobbylm package rather than transformers. The prompt format matters — the model was trained on exactly one layout:

SYSTEM: You are a source-code security auditor. Given a code snippet, decide whether it contains a security vulnerability and reply with one JSON object: {"vulnerable": bool, "cwe": str, "cwe_name": str, "owasp": str}. Use "none" for safe code.
USER: Audit the code below for security vulnerabilities.
Language: C

```c
<your code here>
```
ASSISTANT:

Use detect_prompt() rather than assembling that by hand:

import json, torch, tiktoken
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from hobbylm.config import ModelConfig
from hobbylm.model import MoETransformer
from hobbylm.security_data import detect_prompt, VERDICT_PREFIX, TRUE_ID, FALSE_ID

repo = "rootxhacker/codeastra-500M"
cfg_d = json.load(open(hf_hub_download(repo, "config.json")))
cfg_d.pop("preset", None)
model = MoETransformer(ModelConfig(**cfg_d)).cuda().eval()
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))
enc = tiktoken.get_encoding("gpt2")

code_to_analyze = """
$query = $_GET['query'];
$stmt = $db->prepare($query);
$stmt->execute();
"""

# Fast path: force the verdict position and read one probability (~4ms, no generation)
prompt = detect_prompt(code_to_analyze, "PHP") + VERDICT_PREFIX
ids = torch.tensor([enc.encode_ordinary(prompt)], device="cuda")
with torch.no_grad():
    logits, _ = model(ids)
p_vuln = torch.softmax(logits[0, -1, [TRUE_ID, FALSE_ID]].float(), -1)[0].item()
print(f"P(vulnerable) = {p_vuln:.3f}  ->  {'VULNERABLE' if p_vuln >= 0.3346 else 'safe'}")

Scoring this way is ~26× faster than generating the JSON (4.3 ms vs 110 ms per snippet) and gives you the tunable probability. If you also want the CWE and OWASP labels, generate the completion normally from detect_prompt(code, lang) and parse the JSON with hobbylm.security_data.parse_verdict.

A typical answer:

{"vulnerable": true, "cwe": "CWE-89", "cwe_name": "Improper Neutralization of Special Elements used in an SQL Command (SQL Injection)", "owasp": "A03: Injection"}

Limitations ⚠️

1. Function-level only. It sees a single function, so interprocedural and data-flow vulnerabilities are largely invisible. Access-control bugs needing caller context are its weakest class.
2. Heavily C-weighted training data (92% C). Other languages work but are out-of-distribution — expect lower reliability on Go, Swift, Kotlin and TypeScript.
3. CWE labels confuse related classes. CWE-89 (SQL injection, 85% exact) and CWE-94 (code injection, 82%) are strong; catch-all buckets like CWE-399 and CWE-416 are weak. It frequently finds the right bug and picks a sibling CWE — the detection is better than the label suggests.
4. Multiple vulnerabilities in one snippet are not reliably enumerated; it returns a single verdict.
5. Long code is truncated. Snippets beyond 2048 tokens are head+tail truncated, so the middle of very long functions is unseen.
6. False positives are expected at the recall-oriented thresholds. Results need human verification.
7. It is a 500M research model, not a security product. Use it alongside code review and SAST, not instead of them, and do not gate a release on it alone.

Test Apparatus

All figures come from the held-out test split of ayshajavd/code-security-vulnerability-dataset (17,542 snippets, never trained on), scored with a purpose-built harness that generates the verdict and parses it, plus the calibrated single-forward-pass scorer for threshold-free metrics. The paired evaluation uses 894 vulnerable/patched function pairs drawn from the same held-out split. The base HobbyLM-Chat comparison was run through the identical prompts and harness, so the two columns are directly comparable.

The DeepSeek v4 Flash comparison was run through OpenRouter on the same held-out rows, receiving the identical system/user content in native chat format so it is not penalised for our decoder-only prompt convention, and scored with the same parser and metrics. Reasoning was enabled at low effort: at default effort the model spends ~98% of its output budget thinking and 37% of replies never emit an answer within 8k tokens, which would have measured our token cap rather than its accuracy.

No numbers on this page were copied from another model card — everything here was measured directly. CodeAstra-7B is not comparable to these figures: it was evaluated on a different corpus under a different protocol, so its 83% accuracy and this card's numbers do not sit on the same scale.

Citation 📜

@software{CodeAstra-500M,
  author = {Harish Santhanalakshmi Ganesan},
  title = {CodeAstra-500M: Laptop-Scale Vulnerability Detection},
  year = {2026},
  howpublished = {\url{https://huggingface.co/rootxhacker/codeastra-500M}}
}

License 📄

CodeAstra-500M is released under the Apache License 2.0.

Copyright 2026 [Harish Santhanalakshmi Ganesan]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Acknowledgements 🙏

Thanks to the HobbyLM project for the 500M MoE base model, and to @ayshajavd for compiling the vulnerability dataset this model was trained on.