CodeBERT-Primevul-BigVul is a multi-task model based on Microsoft's codebert-base, fine-tuned to detect vulnerabilities (vul) and classify Common Weakness Enumeration (CWE) types in code snippets. It was developed by mahdin70 and trained on a balanced dataset combining BigVul and PrimeVul datasets. The model performs binary classification for vulnerability detection and multi-class classification for CWE identification.
The model extends codebert-base with two task-specific heads:
The architecture is implemented as a custom MultiTaskCodeBERT class in PyTorch, with the loss computed as the sum of cross-entropy losses for both tasks.
The model was trained on the mahdin70/balanced_merged_bigvul_primevul dataset, which combines:
Splits:
Features:
func: Code snippet (text)vul: Binary label (0 = non-vulnerable, 1 = vulnerable)CWE ID: CWE identifier (e.g., CWE-89) or None for non-vulnerable samplesPreprocessing:
LabelEncoder with 134 unique CWE classes identified across the dataset.The dataset is balanced to ensure a fair representation of vulnerable and non-vulnerable samples, with a maximum of 10 samples per commit where applicable.
The model was trained using the Hugging Face Trainer API with the following arguments:
./logsValidation metrics across epochs:
| Epoch | Training Loss | Validation Loss | Vul Accuracy | Vul Precision | Vul Recall | Vul F1 | CWE Accuracy |
|---|---|---|---|---|---|---|---|
| 1 | 0.4275 | 0.5737 | 0.9519 | 0.7753 | 0.4795 | 0.5925 | 0.0656 |
| 2 | 0.7608 | 0.5450 | 0.9537 | 0.7766 | 0.5133 | 0.6181 | 0.1349 |
| 3 | 0.5624 | 0.5443 | 0.9545 | 0.7669 | 0.5400 | 0.6338 | 0.1749 |
Install the required libraries:
pip install transformers torch datasets huggingface_hub
Below is an example of how to use the model for inference on a code snippet:
from transformers import AutoTokenizer, AutoModel
import torch
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("microsoft/codebert-base")
model = AutoModel.from_pretrained("mahdin70/CodeBERT-Primevul-BigVul", trust_remote_code=True)
model.eval()
# Example code snippet
code = """
bool DebuggerFunction::InitTabContents() {
Value* debuggee;
EXTENSION_FUNCTION_VALIDATE(args_->Get(0, &debuggee));
DictionaryValue* dict = static_cast<DictionaryValue*>(debuggee);
EXTENSION_FUNCTION_VALIDATE(dict->GetInteger(keys::kTabIdKey, &tab_id_));
contents_ = NULL;
TabContentsWrapper* wrapper = NULL;
bool result = ExtensionTabUtil::GetTabById(
tab_id_, profile(), include_incognito(), NULL, NULL, &wrapper, NULL);
if (!result || !wrapper) {
error_ = ExtensionErrorUtils::FormatErrorMessage(
keys::kNoTabError,
base::IntToString(tab_id_));
return false;
}
contents_ = wrapper->web_contents();
if (ChromeWebUIControllerFactory::GetInstance()->HasWebUIScheme(
contents_->GetURL())) {
error_ = ExtensionErrorUtils::FormatErrorMessage(
keys::kAttachToWebUIError,
contents_->GetURL().scheme());
return false;
}
return true;
}
"""
# Tokenize input
inputs = tokenizer(code, return_tensors="pt", padding="max_length", truncation=True, max_length=512)
# Move to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
inputs = {k: v.to(device) for k, v in inputs.items()}
# Get predictions
with torch.no_grad():
outputs = model(**inputs)
vul_logits = outputs["vul_logits"]
cwe_logits = outputs["cwe_logits"]
# Vulnerability prediction
vul_pred = torch.argmax(vul_logits, dim=1).item()
print(f"Vulnerability: {'Vulnerable' if vul_pred == 1 else 'Not Vulnerable'}")
# CWE prediction (if vulnerable)
if vul_pred == 1:
cwe_pred = torch.argmax(cwe_logits, dim=1).item() - 1 # Subtract 1 as -1 is "no CWE"
print(f"Predicted CWE: {cwe_pred if cwe_pred >= 0 else 'None'}")
Vulnerability: Vulnerable
Predicted CWE: 120 # Maps to CWE-120 (Buffer Overflow), depending on encoder
trust_remote_code=True as the model uses custom code from the repository.model.safetensors 499 MBtraining_args.bin 5.2 kBmicrosoft/codebert-base