graphcodebert-code-clone-detection
Binary code-clone detection. Full fine-tune of
microsoft/graphcodebert-base on
PoolC/1-fold-clone-detection-600k-5fold,
using GraphCodeBERT's data-flow-aware pairwise architecture.
Output labels: 0 = not clone, 1 = clone.
Architecture
Not a generic sequence-pair classifier. The two snippets are encoded
separately by one shared GraphCodeBERT encoder, each with its own
graph-guided masked attention, and the two <s> vectors are concatenated for
classification:
Linear(2 x 768 -> 768) -> tanh -> Linear(768 -> 2)
Per-snippet input layout (length 640):
| segment | length | content | position_idx |
|---|---|---|---|
| code tokens | 512 | <s> + BPE code tokens + </s> |
2 .. n+1 |
| data-flow nodes | 128 | one slot per DFG variable node (<unk> id) |
0 |
| padding | remainder | <pad> |
1 |
A data-flow node's input embedding is the average of the embeddings of the
code tokens it was identified from. Graph-guided attention allows: code to
code; <s>/</s> to everything; node to the code tokens it comes from (and
back); node to adjacent nodes.
Preprocessing
The dataset contains Python snippets, so data flow is extracted with the
tree-sitter-python grammar via a port of GraphCodeBERT's DFG_python
extractor (comment/docstring stripping -> AST -> variable states ->
comesFrom / computedFrom edges).
code_length |
512 |
data_flow_length |
128 |
| total sequence length | 640 |
| distinct snippets featurised | 44,950 |
| mean data-flow nodes / snippet | 44.22 |
| snippets with empty data flow | 263 |
| total data-flow edges | 2,481,388 |
| extraction status counts | {"ok": 44930, "comment_strip_failed": 13, "dfg_failed": 7} |
No example was dropped: a snippet whose data flow could not be extracted is kept with an empty graph and counted above.
Data splits
The repository provides one of 5 predefined folds as train + val; those groups are disjoint and are kept as-is. val is partitioned further into validation/test along problem-group boundaries.
similar equals (code1_group == code2_group) for every row, so the group
columns are a perfect label proxy and are never used as features.
| split | source | pairs | positives | negatives | groups |
|---|---|---|---|---|---|
| train | train fold |
50,000 | 25,000 | 25,000 | 240 |
| validation | half of val by group |
20,000 | 10,000 | 10,000 | 29 |
| test | other half of val by group |
20,000 | 10,000 | 10,000 | 30 |
Train/validation/test share no problem group and no code snippet; this is asserted at runtime before training starts. 337,398 pairs of the held-out fold were dropped because their two snippets fell on opposite sides of the validation/test group boundary.
Class weighting: Measured majority-class share 0.5000 is within the 0.6 threshold, so weighted cross entropy is NOT used.
Training
| optimizer | adamw_torch |
| learning rate | 2e-05 |
| scheduler | linear with 0.1 warmup ratio (938 steps) |
| epochs | 3.0 |
| per-device batch size | 16 |
| gradient accumulation | 1 |
| effective batch size | 16 |
| weight decay | 0.01 |
| gradient clipping | 1.0 |
| mixed precision | fp16 |
| gradient checkpointing | False |
| seed | 42 |
| trainable parameters | 125,236,994 |
| training time | 1.923 h |
| GPU | NVIDIA GeForce RTX 5060 Ti (15.9 GB) |
| torch / transformers | 2.11.0+cu128 / 5.17.0 |
Checkpoint selection: best validation F1 (load_best_model_at_end=True,
metric_for_best_model="f1"). Best validation F1 = 0.8672.
The test split was scored once, after selection.
Results
| split | accuracy | precision | recall | F1 | TP | TN | FP | FN |
|---|---|---|---|---|---|---|---|---|
| validation | 0.8557 | 0.8032 | 0.9422 | 0.8672 | 9,422 | 7,692 | 2,308 | 578 |
| test | 0.8747 | 0.8410 | 0.9240 | 0.8805 | 9,240 | 8,253 | 1,747 | 760 |
Test confusion matrix ([[TN, FP], [FN, TP]]): [[8253, 1747], [760, 9240]]
Usage
This checkpoint uses a custom pairwise head and a graph-guided attention
mask, so AutoModelForSequenceClassification will not reproduce these
results. Use the repository's own model class and preprocessing:
import torch
from transformers import AutoTokenizer
from modeling import load_model # from this project
from preprocess import build_snippet_features, CloneCollator
from config import Config
cfg = Config()
tokenizer = AutoTokenizer.from_pretrained("thealper2/graphcodebert-code-clone-detection")
model = load_model("thealper2/graphcodebert-code-clone-detection").eval()
features = build_snippet_features(cfg, [code_a, code_b], tokenizer, num_proc=1)
collator = CloneCollator(features)
batch = collator([(0, 1, 0)]) # (snippet_a, snippet_b, dummy label)
with torch.no_grad():
logits = model(**{k: v for k, v in batch.items() if k != "labels"}).logits
label = int(logits.argmax(-1)) # 0 = not clone, 1 = clone
Limitations
- Trained on competitive-programming Python solutions grouped by problem; "clone" therefore means solves the same problem, which is closer to semantic (Type-4) similarity than to syntactic copy-paste detection.
- Data flow is extracted with the Python grammar only. Other languages need the
matching
tree_sitter_<lang>grammar andDFG_<lang>function. - Snippets longer than 512 BPE tokens are truncated; 885 of 44,950 distinct snippets hit that limit.
- Both directions of a pair are not explicitly symmetrised; the head sees
concat(<s>_1, <s>_2)in the given order.