SkillJev: CV-evidence → skill / evidence / proficiency classifier

SkillJev scores how well candidate evidence (CV excerpts, interview notes) supports a single job requirement. It has three heads:

Head	Labels
skill group	`lang_core, api_design, data_storage, system_design, cloud_devops, reliability_obs, security, testing_quality, frontend, leadership`
evidence	`supported, partial, contradicted, insufficient`
proficiency	`0, 1, 2, 3, 4, 5, unknown` (unknown ≠ 0: level 0 is used only when the evidence explicitly says the candidate has no experience)

Base model: AlexWortega/openjev, subfolder qwen3.5-4b-nli-v5 (OpenJev 4B, Qwen3.5 NLI).
Adapter: LoRA r=16, α=32, all linear layers.
Heads: 3 linear heads on the last-token hidden state (heads.pt).
Training: 1 epoch-slice, bf16, lr 1e-4, batch 4, max length 192, 500 steps on Colab L4.
Training data: niko0xdev/skilljev-data (v008).
Labeling standard: RUBRIC.md, including the Real-CV labeling convention v0.2.

Results: locked held-out set (n = 90, never trained on)

The held-out set has 70 synthetic samples built from held-out-only templates and 20 human/agent-reviewed real Djinni CV excerpts.

Model	skill	evidence	prof exact	prof ±1	ECE (ev / prof)
tasksource/ModernBERT-base-nli, zero-shot	73.3	56.7	58.9	63.3	–
OpenJev 4B, zero-shot	88.9	68.9	71.1	80.0	–
SkillJev (this checkpoint: v008, step 500)	100.0	97.8	95.6	96.7	0.023 / 0.033
SkillJev v008, step 1000 (dev-selected)	100.0	97.8	94.4	98.9	0.024 / 0.038

ECE is measured after temperature scaling fit on the dev split (T: skill 0.5, evidence 2.4, proficiency 2.8).

Caveat: step 500 was chosen because of its held-out score. The checkpoint picked purely on dev (step 1000) reaches 94.4% proficiency exact. The 1.2-point gap is one sample out of 90, so treat ~94–96% as the realistic range. The held-out set is small (n=90), and every number carries roughly ±3–5 points of sampling noise.

Usage

# pip install "transformers>=4.57" peft accelerate  (GPU with ~12 GB)
# see inference.py in this repo
from inference import predict
predict("Strong PostgreSQL skills", [("cv","experience","Partitioned a 3TB PostgreSQL table and tuned indexes, cutting queries from 40s to 1s.")])
# {'skill_group': {'label': 'data_storage', ...}, 'evidence': {'label': 'supported', ...}, 'proficiency': {'label': '4', ...}}

Input format: Requirement: <text>\nEvidence:\n[<source_type>/<section>] <text> (one line per evidence item, or (no evidence)).

Intended use & limitations

This is a decision-support tool for structuring CV screening. It is not an automated hiring decision. Keep a human in the loop and route low-confidence (<0.7), contradicted, or level ≥4 CV-only predictions to review.
Coverage: English, IT/software roles, 55 skills in 10 groups (TAXONOMY.md). Other domains or languages are out of distribution.
Most training data is template-generated synthetic text. The real-CV portion is about 1.2k agent-reviewed excerpts plus about 5.7k rule-labelled (silver) excerpts, so expect lower accuracy on unusual phrasing.
Proficiency is judged from a single excerpt. It reflects the evidence given, not the person's true skill. CV self-reports are not verified.
Possible biases come from the Djinni (Ukrainian job market) CV distribution and from the labeling convention itself.

MLX / Apple silicon

The release is standard PEFT safetensors plus a small PyTorch heads.pt. To run on MLX: merge the LoRA into the base model (peft merge_and_unload()), convert with mlx_lm.convert, and port the three linear heads (weights in heads.pt) to mlx.nn.Linear. No official MLX build is provided.

License & attribution