F2LLM-v2-1.7B ONNX, 4/8-bit quantized

Quantized ONNX versions of codefuse-ai/F2LLM-v2-1.7B for ONNX Runtime. This is a quantized derivative: the model, its training and its licence (Apache-2.0) belong to its original authors. All 17 variants we built are here, including those below our quality gate, with their measurements.

Recommended: mix50-gptq-sq50-a4, 1,629 MB instead of 6,885 MB, 1,788 MB RAM instead of 5,603 MB; nDCG@10 vs FP32 +0.19 [-0.44, +0.85] points on FreshRet Small, -0.38 [-1.22, +0.28] on MIRACL.

Usage

pip install "onnxruntime>=1.31" tokenizers numpy huggingface_hub
hf download nynefold/F2LLM-v2-1.7B-ONNX inference.py --local-dir .

from inference import load

model = load("nynefold/F2LLM-v2-1.7B-ONNX")  # recommended variant; load("nynefold/F2LLM-v2-1.7B-ONNX", variant="...") for another
q = model.embed_queries(["How do I reset the router?"])
d = model.embed_documents(["Hold the reset button for ten seconds.", "The router has four LAN ports."])
print(q @ d.T)  # cosine similarities (vectors are L2-normalized)

inference.py (in this repo) needs only ONNX Runtime, tokenizers and NumPy. Given a repo id, it downloads the variant as plain files into ~/.cache/nynefold/<repo id> (set NYNEFOLD_CACHE to change the folder); ONNX Runtime does not load model.onnx.data through the Hub cache's symlinks. load("path/to/folder") uses a local copy. To run the graphs another way:

• Inputs input_ids and attention_mask, int64 [batch, seq], right-padded.
• Query prefix "Instruct: Given a question, retrieve passages that can help answer the question.\nQuery: ", no document prefix. At most 512 tokens (the final end token is kept).
• Last-token pooling is inside the graph; output embedding [batch, dim].
• The output is L2-normalized; compare by dot product.
• config.json holds all of this in machine-readable form.

Variants

Each folder holds model.onnx + model.onnx.data and the recipe.json it was built from. Quality deltas are paired against FP32 in nDCG@10 points with 95% bootstrap intervals.

• Recommended variant chosen on FreshRet Small quality; MIRACL is the held-out check (120 dev queries in en, es, fr, de, zh, ar from mteb/MIRACLReranking, MTEB's reranking version of MIRACL, with BM25 top-20 pools). Calibration and stage A use other, disjoint MIRACL dev queries.
• "Below quality gate": stage A fidelity under the gate, or more than 1.0 nDCG@10 point lost on FreshRet Small. "Passes stage A only": not measured on FreshRet Small (finalists only), so its retrieval quality is unconfirmed.
• RAM: peak process memory while embedding one 256-token passage, in a process with only ONNX Runtime, tokenizers and NumPy loaded.
• a4 variants use INT8 compute (accuracy_level 4) and are meant for CPUs with fast INT8 instructions; a0 variants compute in FP32 from the quantized weights.
• Tested with ONNX Runtime 1.31. Variant names are explained in nynefold/retrieval-quant.

How they were made

Linear layers use ONNX Runtime's MatMulNBits and the token table GatherBlockQuantized, with bits, algorithm and compute mode set per tensor (GPTQ, RTN, clip search; mixed 4/8-bit from a sensitivity scan; SmoothQuant where noted). The method, gates, all results and the pipeline comparison with the other models are in nynefold/retrieval-quant. The evaluation set is nynefold/FreshRet-Small, with mteb/MIRACLReranking for calibration, fidelity and the held-out check; all six repos are in the collection retrieval-quant: quantized ONNX retrievers.

Licence

Apache-2.0, as the base model: see LICENSE. Please cite and credit the original model, codefuse-ai/F2LLM-v2-1.7B.