Pocket Rerank — Ettin 17M ONNX
A fused CPU inference build of the Ettin 17M cross-encoder, paired with an overlapping-window runtime that returns the passage behind each document score. Designed for local English RAG and search reranking without a PyTorch runtime.
Preview release. This project did not train the original weights. It fuses the upstream encoder and scoring modules into one dynamic ONNX graph and provides document-window handling, evidence offsets, a demo/API, and reproducible evaluation. It is not an official Sentence Transformers release and does not claim state-of-the-art quality.
Install and search your own documents
The 0.2.0 toolkit adds local corpus search, source IDs and a command-line interface. The model weights are unchanged. Install the small wheel, then explicitly download only the model files and examples you need (Python 3.11+):
pip install 'pocket-rerank[search] @ https://huggingface.co/samurvivor-07/pocket-rerank-ettin-17m-onnx/resolve/v0.2.0/dist/pocket_rerank-0.2.0-py3-none-any.whl'
hf download samurvivor-07/pocket-rerank-ettin-17m-onnx --revision v0.2.0 --include 'model.onnx' 'tokenizer.json' 'config.json' 'examples/*' --local-dir pocket-model
pocket-rerank --model-dir pocket-model --corpus pocket-model/examples/documents.jsonl --query 'How do I rotate an API key without downtime?' --top-k 2
Full quickstart and custom retriever integration · Demo test instructions · Live browser demo
The search command preserves your document IDs, metadata and evidence offsets. For existing vector/search results, use the lightweight reranker directly:
from pocket_rerank import PocketReranker
ranker = PocketReranker("pocket-model")
results = ranker.rank("How do I rotate an API key without downtime?", [
"API keys identify clients.",
"Create a second key, deploy it to clients, verify traffic, then revoke the old key.",
])
print(results[0])
No PyTorch is needed for inference. This is a custom ONNX runtime, not a Transformers
AutoModel checkpoint. It never executes Hub remote model code. Initial installation
and downloads need network access; subsequent inference runs locally. The Space
runs inference in your browser and has no hosted Python API. Browser evidence is
token-decoded; the evaluated Python runtime returns original character offsets.
The full source repository also contains the optional local FastAPI/Gradio app. After downloading the full repository and entering its directory:
pip install '.[server]'
MODEL_DIR=. uvicorn app:app --host 127.0.0.1 --port 7860
Files and provenance
model.onnx: full-precision fused encoder and score head, 67,654,196 bytes.tokenizer.json: upstream tokenizer, 3,583,327 bytes.config.json: runtime settings and immutable upstream revision.pocket_rerank/: Python runtime and evidence-window implementation.reports/: conversion checks and SciFact evaluation results.SHA256SUMS.json: file integrity hashes.LICENSEandNOTICE: Apache 2.0 and attribution.
Upstream: cross-encoder/ettin-reranker-17m-v1
at commit 9e4aa35321a6dd1a43ca313f500c4b4f7cfb5cc6.
No fine-tuning, additional training data, or new capability training was performed.
The model is not independent of the upstream checkpoint's limitations.
Evaluation
On the 300-query SciFact test split, NDCG@10 was 0.7212 for FP32 windows,
0.7160 for FP32 head-only truncation, and 0.6519 for BM25 alone.
Observed median reranking latency for 20 candidates was 680 ms with windows and
361 ms with head-only truncation, using two CPU threads on local Apple Silicon.
The window-minus-head difference is 0.00523, with a paired bootstrap 95% interval
of [-0.01409, 0.02543]; this does not establish a reliable improvement over truncation.
The window-minus-BM25 interval is [0.04162, 0.09830]. These are descriptive intervals
from 10,000 paired query resamples, without multiplicity adjustment. See
reports/uncertainty.json. Timing was not isolated from other local work.
INT8 windows scored 0.7081 at a 428 ms median and are excluded from this release.
See reports/scifact.json for full results across all 300 SciFact test queries,
candidate selection, dataset commits, per-query metrics, latency, and limitations.
Candidates are the top 20 BM25 results from the full 5,183-document corpus;
relevant documents are never injected. The protocol compares BM25, 256-token
head truncation, and 256-token overlapping windows. NDCG@10 is computed against
all positive test relevance judgments. Results are not comparable to the upstream
MTEB headline benchmark, and training overlap has not been audited.
On a separate 12-case synthetic tail-evidence regression probe, full-precision windowing retrieved the expected document first in 5/12 cases, versus 0/12 with head-only truncation. This deliberately narrow probe is not a general accuracy score. The ONNX graph matched upstream FP32 scores within 0.000004 on the tested pairs of length 24–512 tokens. These checks are limited, not a proof over all inputs. The experimental INT8 graph was evaluated but is not part of this release.
Compact alternatives, matched protocol
We also compared three native CrossEncoder models using identical BM25 top-20 candidates and 256-token head truncation. On SciFact (300 test queries), NDCG@10 was 0.71601 for Ettin 17M, 0.72326 for Ettin 32M, and 0.68003 for MiniLM-L6-v2. On NFCorpus (323 test queries), results were 0.31834, 0.32573, and 0.33212 respectively. MiniLM performed better on NFCorpus, so these results do not establish a universal winner. Ettin 32M's paired 95% gain interval includes zero on both datasets.
These head-only comparisons are distinct from the ONNX windowing results above.
See the comparison protocol and uncertainty, exact model/data
revisions in evaluation/comparison_sources.json, and per-query JSON reports.
The authored developer diagnostic set includes known failures and is not a held-out
accuracy benchmark. Toolkit 0.2.0 retains the original 17M weights.
Reproduce
The repository includes the export and evaluation source in scripts/, the source
revision in upstream_revision.json, and development-lock.txt from the measured
environment. Install the development extras with pip install '.[dev]', then run
python scripts/fetch_model.py, python scripts/export_model.py,
python scripts/benchmark.py, python scripts/fetch_benchmark.py,
python scripts/evaluate_scifact.py, and python scripts/analyze_uncertainty.py.
The fetch steps require network access; evaluation runs locally afterward.
Browser verification
Hosted browser inference passed the key-rotation smoke case. Across nine short pairs in three examples, browser scores differed from Python
by at most 0.000006. This is a limited numerical check, not full browser parity. A longer local browser case inspected four windows and
returned the tail instruction. These are smoke checks, not browser benchmark parity. See reports/browser-score-parity.json.
The included HTTP 429 example failed: an authentication-token document ranked above
the correct Retry-After/backoff advice. See reports/browser-verification.json.
Limits
English relevance ranking only. Scores are uncalibrated and do not establish truthfulness, answer correctness, or document safety. Maximum-window aggregation can favor long documents. Document text is evidence for the score, not a generated explanation. A first-stage retriever must already supply relevant candidates.
Default pair length 256 tokens, overlap 48; supported lengths 160–512. Queries over 128 tokens and requests beyond the runtime's document/window budget fail explicitly. CPU throughput and latency depend on hardware, passage length and batching; measurements here are from local Apple Silicon, not the hosted Space.
License
Apache License 2.0. Retain the upstream attribution and notices when redistributing. SciFact was used for evaluation under CC BY-SA 4.0; no dataset text is distributed in this repository. All original model credit belongs to its upstream authors.