INF-Retriever-v1-1.5B-Tool
Model Overview
INF-Retriever-v1-1.5B-Tool is a tool-retrieval-specialized version of INF-Retriever-v1-1.5B, the LLM-based dense retrieval model developed by INF TECH. It is fine-tuned on the ToolRet training data mixed with large-scale synthesized tool-retrieval queries, adapting the strong general retrieval capability to the tool-use scenario for LLM agents.
On the official ToolRet benchmark (7,961 queries over a 44,453-tool corpus, w/ instruction protocol), it scores NDCG@10 = 0.509 / Recall@10 = 0.609, surpassing the previous best embedding model on the leaderboard (0.4596 NDCG@10) while staying on par with its own zero-shot score — confirming the recipe adapts to the domain without degrading the base model.
Key Features
- Tool-retrieval specialized: fine-tuned with real user queries from the official ToolRet training set plus LLM-synthesized scene-style queries covering the full tool corpus.
- No degradation on a strong base: the hybrid recipe (official queries + full-coverage synthetic queries with hard negatives) preserves the base model's zero-shot strength.
- Full-corpus hard negatives: same-family tool confusion + cross-family BM25 lexical-confusion negatives.
Training data (full disclosure)
| Source | Pairs | Note |
|---|---|---|
| ToolRet-Training-20w (shard 0, official) | 34,676 | positives anchored to the 44,453-tool corpus; test-query contamination filter applied (129 exact matches removed) |
| Synthesized queries (ours) | 87,747 | LLM scene-style query generation per tool doc, lexical-overlap gate (query↔positive Jaccard ≤ 0.35), hard negatives mined per strategy below |
Training corpus = the full official 44,453-tool corpus. Test queries untouched.
Fine-tuning hyperparameters: lr 1e-5 (constant), 1 epoch, effective batch 64, 7 negatives per query, full-parameter, DeepSpeed ZeRO-2, bf16.
Usage
Sentence Transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("infly/inf-retriever-v1-1.5b-tool", trust_remote_code=True)
model.max_seq_length = 512
queries = [
"Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: Given a `stock market data` task, retrieve tools that can provide straddle data for a given stock ticker. What is the straddle data for Alphabet Inc. (GOOGL)?",
]
documents = [
'{"name": "straddle", "description": "This function retrieves the straddle data for a given stock ticker from the RapidAPI stock and options trading data provider.", ...}',
]
query_embeddings = model.encode(queries)
document_embeddings = model.encode(documents)
scores = (query_embeddings @ document_embeddings.T) * 100
print(scores.tolist())
Transformers
import torch
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel
def last_token_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor:
left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
if left_padding:
return last_hidden_states[:, -1]
sequence_lengths = attention_mask.sum(dim=1) - 1
batch_size = last_hidden_states.shape[0]
return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [f'Instruct: {task}\nQuery: I need to check today\'s currency conversion rates before an international payment']
documents = ['{"name": "currency_convert", "description": "Converts an amount from one currency to another using the latest exchange rates.", ...}']
input_texts = queries + documents
tokenizer = AutoTokenizer.from_pretrained('infly/inf-retriever-v1-1.5b-tool', trust_remote_code=True)
model = AutoModel.from_pretrained('infly/inf-retriever-v1-1.5b-tool', trust_remote_code=True)
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:1] @ embeddings[1:].T) * 100
print(scores.tolist())
Instruction format
The model keeps the base model's query instruction unchanged (the one used in training and in all reported ToolRet numbers):
Instruct: Given a web search query, retrieve relevant passages that answer the query
Query: {ToolRet task instruction + user query}
ToolRet task-specific instructions (e.g. "Given a bacterial population prediction task, ...") are included as part of the query text, following the official benchmark format. Documents are encoded without any prefix.
Evaluation
ToolRet Benchmark (ACL 2025)
Full 44,453-tool corpus, FAISS inner-product retrieval, pytrec_eval, micro-average over all 7,961 test queries, w/ instruction protocol:
| Model | NDCG@10 | Recall@10 | Recall@100 |
|---|---|---|---|
| INF-Retriever-v1-1.5B-Tool | 0.509 | 0.609 | 0.753 |
| INF-Retriever-v1-1.5B (zero-shot) | 0.511 | 0.620 | 0.805 |
| gte-qwen2 (official training, leaderboard best embedding) | 0.4596 | — | — |
| BM25 (our harness) | 0.383 | 0.475 | — |
Leaderboard submission: mangopy/tool-retrieval-benchmark#21
Citation
If you use this model, please cite the ToolRet benchmark:
@inproceedings{pyatkin2025toolret,
title={Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models},
booktitle={Findings of ACL 2025},
year={2025}
}