sciseek-micro is a highly efficient embedding model for scientific literature search. It is fine-tuned specifically for embedding search queries and title + abstract descriptions of research papers. We release it alongside the even smaller sciseek-nano; both are trained on the sciseek-train dataset.
On SciSeek, both models are Pareto efficient regarding their model and embedding size, outperforming much larger models. sciseek-micro reaches the highest score of all models we evaluated on the SciSeek benchmark.
| model | base model | params | dim | SciSeek | LitSearch | 8-bit ONNX |
|---|---|---|---|---|---|---|
| sciseek-nano | granite-embedding-small-english-r2 | 47M | 384 | 0.646 | 0.512 | 53 MB |
| sciseek-micro (this model) | granite-embedding-english-r2 | 149M | 768 | 0.715 | 0.584 | 167 MB |
nDCG@10 on SciSeek (3,010 machine-generated queries over 200,000 arXiv papers) and LitSearch (597 real researcher questions).
The models run in sentence-transformers, ONNX Runtime and transformers.js, and we provide docker compose files for deploying the models with TEI and vLLM.
Embed papers as "Title:\n{title}\nAbstract:\n{abstract}"; queries need no prefix or instruction. The CLS token is used for pooling; L2-normalise and compare with the dot product. The model is specialised for literature search; for other tasks, use a general model such as granite-embedding-english-r2, the base model of sciseek-micro.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("daniel-gomm/sciseek-micro")
query = "methods for compressing large language models via knowledge distillation"
papers = [
"Title:\nTinyBERT: Distilling BERT for Natural Language Understanding\n"
"Abstract:\nWe propose a novel Transformer distillation method...",
"Title:\nObservations of Type Ia supernovae in distant galaxies\n"
"Abstract:\nWe report photometric measurements...",
]
q = model.encode(query, normalize_embeddings=True)
d = model.encode(papers, normalize_embeddings=True)
print(q @ d.T) # cosine similarity
In the browser or Node with transformers.js:
import { pipeline } from '@huggingface/transformers'
const extractor = await pipeline('feature-extraction', 'daniel-gomm/sciseek-micro', {
device: 'webgpu', dtype: 'fp16', // or device: 'wasm', dtype: 'q8' (167 MB)
})
const q = await extractor('methods for compressing large language models via knowledge distillation',
{ pooling: 'cls', normalize: true }) // q.dims = [1, 768]
Not every browser exposes WebGPU with fp16, so fall back to WASM if loading fails, and run the extractor in a Web Worker. For a server, deploy/ has docker compose files for Text Embeddings Inference and vLLM; vLLM returns unnormalised embeddings, so normalise them yourself.
We evaluated 14 open embedding models (23M to 4B parameters) and BM25 in the same setup. We use the query/document prompts and document format specified in each of the model cards. Search is implemented as the exact cosine over the full corpus. No setting was tuned on SciSeek or LitSearch, and every top-10 result of every run on SciSeek has a relevance label.
| model | params | dim | SciSeek | LitSearch |
|---|---|---|---|---|
| sciseek-micro (this model) | 149M | 768 | 0.715 | 0.584 |
| Qwen3-Embedding-4B (8-bit) | 4.0B | 2560 | 0.682 | 0.598 |
| sciseek-nano | 47M | 384 | 0.646 | 0.512 |
| granite-embedding-english-r2 | 149M | 768 | 0.623 | 0.497 |
| jina-embeddings-v5-text-small | 596M | 1024 | 0.620 | 0.556 |
| Qwen3-Embedding-0.6B | 596M | 1024 | 0.597 | 0.538 |
| jina-embeddings-v5-text-nano | 239M | 768 | 0.587 | 0.541 |
| granite-embedding-small-english-r2 | 47M | 384 | 0.587 | 0.486 |
| gte-modernbert-base | 149M | 768 | 0.581 | 0.501 |
| mdbr-leaf-ir | 23M | 768 | 0.554 | 0.490 |
| embeddinggemma-2 | 270M | 768 | 0.550 | 0.481 |
| bge-small-en-v1.5 | 33M | 384 | 0.499 | 0.428 |
| BM25 | — | — | 0.471 | 0.394 |
| all-MiniLM-L6-v2 | 23M | 384 | 0.468 | 0.355 |
| SPECTER2 | 110M | 768 | 0.297 | 0.248 |
| queries | 3,010 | 597 |
nDCG@10 at each model's native embedding size. Qwen3-Embedding-4B was run with 8-bit weights to fit an 8 GB GPU. Exact settings and run files can be found in the SciSeek benchmark under baselines/systems.json.
Compared to the base model, fine-tuning has the largest effect on descriptive queries (questions, problem descriptions, citation contexts, literature reviews) where the semantic context is more relevant for determining relevance than direct overlap. Looking up the titles of known items, which BM25 already solves, drops slightly for both models.
Other scientific retrieval sets. The fine-tuning keeps the base model's general ability on most sets (nDCG@10, change against the base model with 95% CI; * = CI excludes 0):
| benchmark | what it tests | n | sciseek-nano | sciseek-micro |
|---|---|---|---|---|
| DORIS-MAE | complex multi-aspect literature queries | 100 | 0.651 (+0.003) | 0.630 (−0.028*) |
| BEIR TREC-COVID | biomedical literature search | 50 | 0.630 (−0.035*) | 0.712 (−0.017) |
| BEIR SciFact | claim verification | 300 | 0.741 (+0.001) | 0.748 (−0.001) |
| BEIR NFCorpus | lay-language nutrition/medical queries | 323 | 0.366 (−0.004) | 0.370 (−0.008) |
| BEIR SciDocs | a paper title as the query, citing papers as targets | 1,000 | 0.229 (−0.010*) | 0.240 (−0.011*) |
| build | file | sciseek-nano | sciseek-micro |
|---|---|---|---|
| PyTorch | model.safetensors |
95 MB | 298 MB |
| ONNX fp32 | onnx/model.onnx |
191 MB | 597 MB |
| ONNX fp16 (WebGPU) | onnx/model_fp16.onnx |
96 MB | 298 MB |
ONNX 8-bit (WASM, q8) |
onnx/model_quantized.onnx |
53 MB | 167 MB |
The fp32 and fp16 ONNX builds give the same embeddings as PyTorch (minimum cosine 1.0000 on 256 texts). The 8-bit builds do not significantly decrease performance and achieve the same test results on both benchmarks. On SciSeek, nDCG@10 changes by −0.000 [−0.001, +0.001] (nano) and −0.001 [−0.001, +0.000] (micro), on LitSearch it changes by +0.004 [+0.000, +0.007] and +0.001 [−0.002, +0.004].
sciseek-micro is fine-tuned from granite-embedding-english-r2 using the sciseek-train dataset, which consists of semi-synthetic training data sourced from arXiv publications and synthetic search queries. We train with InfoNCE with in-batch and mined hard negatives and false-negative masking, with 512 papers per step for one epoch. We select the best performing model from a separate validation set and blend the fine-tuned and base weights with WiSE-FT (1 - α) * base + α * fine-tuned with α = 0.6 at step 3,248 of 3,248.
We ensure that the training data is entirely independent of all papers from the evaluation benchmarks SciSeek, LitSearch, and DORIS-MAE to ensure there is no leakage.
We will provide full details of the training process in an upcoming paper.
The SciSeek embedding models are trained for literature search, thus the performance for generic retrieval tasks is not sufficiently assessed. Our training and evaluation are also limited to papers sourced from arXiv, which limits the domain of documents the model is trained on, primarily consisting of physics, mathematics, and computer science papers. Other research domains are covered to a lesser degree. Additionally, the training data is semi-synthetic and queries are generated. While evaluation results suggest generalization to human-written queries, we expect a narrower query distribution in our training data than in real search.
Apache-2.0. The model builds on IBM Granite Embedding R2. Thank you to arXiv for use of its open access interoperability.
@misc{gomm2026sciseek,
title = {SciSeek: Benchmarking and Training Embedding Models for Scientific Literature Search},
author = {Gomm, Daniel},
year = {2026},
url = {https://huggingface.co/datasets/daniel-gomm/SciSeek}
}
onnx/model.onnx 597 MBonnx/model_fp16.onnx 298 MBmodel.safetensors 298 MBonnx/model_quantized.onnx 167 MB