skillrep-reranker-0.6b

A cross-encoder for agent-skill retrieval: given a task description and a candidate skill document, judge whether the skill helps complete the task. Fine-tuned from Qwen/Qwen3-Reranker-0.6B.

It reorders the candidates recalled by skillrep-embedding-0.6b. Scoring is one forward pass per (task, skill) pair, so run it on a shortlist, the encoder's top 10, not on the whole registry. Skill documents follow the schema of skillcorpus-demo-1k.

Requires transformers>=4.56 (the dtype= argument was named torch_dtype= before that).

Usage

The model answers a yes/no question, and the relevance score is the difference between the yes and no logits at the final position. sigmoid of this score is P("yes") and gives the same ordering.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "EverMind-AI/skillrep-reranker-0.6b"
tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16).cuda().eval()

PREFIX = ('<|im_start|>system\nJudge whether the Document meets the requirements '
          'based on the Query and the Instruct provided. Note that the answer can '
          'only be "yes" or "no".<|im_end|>\n<|im_start|>user\n')
SUFFIX = '<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
INSTRUCT = ("Given a coding task description, judge whether the skill document "
            "is relevant and useful for completing the task")

YES, NO = tok.convert_tokens_to_ids("yes"), tok.convert_tokens_to_ids("no")
PRE = tok.encode(PREFIX, add_special_tokens=False)
SUF = tok.encode(SUFFIX, add_special_tokens=False)


def pair(task, skill):
    doc = f"{skill['name']} | {skill['description'][:500]} | {skill['body'][:2000]}"
    return f"<Instruct>: {INSTRUCT}\n\n<Query>: {task}\n\n<Document>: {doc}"


def score(task, skills, max_length=4096):
    """One score per skill; higher means more relevant."""
    budget = max_length - len(PRE) - len(SUF)       # truncate the pair, never the template
    ids = [PRE + tok.encode(pair(task, s), add_special_tokens=False,
                            truncation=True, max_length=budget) + SUF for s in skills]
    batch = tok.pad({"input_ids": ids}, return_tensors="pt").to(model.device)
    with torch.no_grad():
        logits = model(**batch).logits[:, -1, :]
    return (logits[:, YES] - logits[:, NO]).float().tolist()


task = "resolve conflicts after a git merge"
skills = [
    {"name": "resolve-conflicts", "description": "Resolve git merge conflicts.",
     "body": "# Resolving merge conflicts\n1. Run `git status` to list conflicted files ..."},
    {"name": "sourdough", "description": "Bake sourdough bread.",
     "body": "# Sourdough\nMix flour, water and starter ..."},
]
print(score(task, skills))
# -> [3.0, -4.06]

Exact scores shift in the last decimal with dtype and hardware; the ordering is what matters.

Keep the PREFIX / SUFFIX template and the <Instruct> / <Query> / <Document> layout, with blank lines between the parts, exactly as above. The score is read off the final-position logits, so truncate the pair text, not the template.

Reranking a shortlist

# candidates: the top 10 skills that skillrep-embedding-0.6b returns for `task`
scores = score(task, candidates)
reranked = [c for _, c in sorted(zip(scores, candidates), key=lambda x: -x[0])]

Via the chat template

chat_template.jinja reproduces the prompt string above, so the tokenizer can assemble it instead. It reads three roles: system carries the instruction (omit it to get the default shown above), query the task, and document the skill:

prompt = tok.apply_chat_template([
    {"role": "query",    "content": task},
    {"role": "document", "content": "resolve-conflicts | Resolve git merge conflicts. | ..."},
], tokenize=False)

Input lengths

Intended use

Second-stage reranking of the top 10 candidates that skillrep-embedding-0.6b recalls for a task. Treat the output as a ranking signal rather than a calibrated probability: compare scores within one candidate list, not across different tasks.

License

Apache-2.0, inherited from the base model. Skills keep their own upstream licenses.