Q8A64 model file for VelociQ8
Fast CPU inference for GGUF models on Arm and x86
Embeddings, dense and MoE LLMs
Runtime · Releases · Usage · All models

multilingual-e5-base · Q8A64

This repository holds multilingual-e5-base converted to Q8A64, the 8-bit weight format of VelociQ8, to run on the CPU. It turns text in about 100 languages into vectors for retrieval, similarity, classification and clustering, with a short prefix on queries and on passages.

This file loads only in VelociQ8 1.1 or later. Other GGUF tools, including standard llama.cpp, cannot load Q8A64 files. The automatic snippets that Hugging Face shows under Use this model (for llama.cpp, Ollama and similar tools) do not work with these files.

Get the runtime

VelociQ8 is free-to-use proprietary software built on llama.cpp. It ships the usual tools (llama-server, llama-cli, llama-embedding, llama-bench and llama-quantize) with the same options and the same HTTP API, including the OpenAI-compatible endpoints. Download the archive for your machine from the table (GitHub release) or from the runtime folder of this repository, which has the same files. Check it against SHA256SUMS from the same place.

The Ollama archives are a portable Ollama with VelociQ8 inside, and do not touch an installed Ollama. The x86-64 ones also use an NVIDIA, AMD or Intel GPU through Vulkan when its driver is installed; see Ollama. The Linux archives need glibc 2.28 or newer. There is also a Docker image for Linux on Arm and x86-64, ghcr.io/dhaern/velociq8 (see Docker). The other archives and the Docker image run on the CPU. VelociQ8 tools cannot download models: download the file from this repository first and pass it with -m.

NVIDIA GPUs (CUDA)

For NVIDIA GPUs there are also portable Ollama archives with CUDA. They are larger (about 1 GB), so they are only on GitHub: velociq8-1.5-ollama-0.40.13-windows-x64-cuda.zip and velociq8-1.5-ollama-0.40.13-linux-x64-cuda.tar.gz. They need an NVIDIA GPU of the Turing generation or newer (GeForce RTX 20 series onwards) with a current driver.

Questions about this file go to the Community tab; problems with the runtime, to GitHub issues.

Files

SHA256 of the weights file, in the format sha256sum -c reads:

6d5cfd1e7dd4ad72d49b34af95c9ea1e84285e6d1e843d804599fc2e669e80c7  multilingual-e5-base-Q8A64.gguf

Quick start

Unpack the archive that matches your CPU (Linux Arm64 shown) and download the file:

tar xf velociq8-1.5-linux-arm64.tar.gz && cd velociq8-1.5-linux-arm64
REPO=VelociQ8/multilingual-e5-base-Q8A64-GGUF
FILE=multilingual-e5-base-Q8A64.gguf
curl -L -O https://huggingface.co/$REPO/resolve/main/$FILE

You can also download it from the Files tab or with hf download VelociQ8/multilingual-e5-base-Q8A64-GGUF multilingual-e5-base-Q8A64.gguf --local-dir .. On Windows use llama-server.exe from the zip; the options are the same.

Start an embeddings server. Use -t for the number of physical cores you want to give the model:

./llama-server -m multilingual-e5-base-Q8A64.gguf --embedding -t 4 -b 2048 -ub 2048 --port 8080

Each input must fit in one batch: -b 2048 -ub 2048 accepts texts of up to 2,048 tokens (the default, 512, rejects longer ones). For longer texts raise both values, up to the context of the model; memory use grows with them.

Ask for the embeddings of a query and a document in one request (the prefixes are explained under Prompt):

curl -s http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"input": ["query: How do solar panels produce electricity?", "passage: Solar panels use photovoltaic cells to convert sunlight into electrical energy."]}'

The reply has one vector of 768 numbers per input, in the same order. Each vector has unit length, so the dot product of two vectors is their cosine similarity.

Prompt

multilingual-e5-base is used with a fixed prefix in front of the query text and a different one in front of the document text.

A query is sent like this (the prefix is everything before the sample question):

query: How do solar panels produce electricity?

A document is sent like this:

passage: Solar panels use photovoltaic cells to convert sunlight into electrical energy.

Measurements

This model has no measurements of its own. Results for other models are published per model family (embeddings, dense LLMs, mixture-of-experts LLMs) in benchmarks.md (speed and memory) and quality.md (accuracy against full precision).

Verified: loads with VelociQ8 1.5 and passes a basic functional test (2026-10-21).

What was checked, with the VelociQ8 1.5 runtime, on the file in this repository:

This is a functional check on a few short texts, not a quality benchmark. It shows that the converted file gives vectors very close to the original model's; it says nothing about retrieval quality on your own data.

About the base model

Source: the original card of intfloat/multilingual-e5-base, by Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder and Furu Wei (technical report). What follows is copied or condensed from it and is their statement, not VelociQ8's; it leaves out the usage examples. Read on 2026-10-21. If anything here differs from the original, the original is right. Benchmark tables are not copied: the Mr. TyDi table is in the original card, and the card points to unilm/e5 to reproduce its BEIR and MTEB results.

Description

Multilingual-E5-base is a multilingual text embedding model described in Multilingual E5 Text Embeddings: A Technical Report. It is initialized from xlm-roberta-base and trained in two stages: contrastive pretraining with weak supervision, then supervised fine-tuning. For labeled data sets only their training sets are used.

Architecture and size

• 12 layers, 768-dimension vectors.
• Initialized from xlm-roberta-base; the vector of a text is the mean of its token vectors, normalized.

Languages

About 100 languages, those of xlm-roberta. The card warns that low-resource languages may see lower performance.

Intended use

Passage retrieval in open QA and ad-hoc information retrieval, semantic similarity, bitext mining, paraphrase retrieval, and embeddings as features for classification and clustering.

Limitations and risks

• Long texts are truncated to at most 512 tokens.
• Low-resource languages may see lower performance.
• The cosine similarity scores fall mostly between 0.7 and 1.0, because training used a low temperature (0.01) in the InfoNCE contrastive loss. The card says that what matters is the relative order of the scores, not their absolute values.

Prompt format

Inputs must start with query: or passage: , even for non-English text; without them performance drops. The card's rules of thumb:

• Use query: and passage: for asymmetric tasks such as passage retrieval in open QA and ad-hoc information retrieval.
• Use query: for symmetric tasks such as semantic similarity, bitext mining and paraphrase retrieval.
• Use query: when the vectors are features, for example for linear-probe classification or clustering.

License

The card gives the license only in its metadata (license: mit). It gives no copyright line, and the original repository has no LICENSE file.

Citation

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672},
  year={2024}
}

Licenses

Two licenses apply, and they do not mix:

• Runtime: VelociQ8 Software License Agreement. It covers the VelociQ8 software only, that is, the archives in the releases. The runtime is free to use for personal or commercial work.
• Weights: the license of the original model. The weights in this repository are a converted copy of the model described in About the base model. They stay under the original license, stated below. The VelociQ8 license agreement does not cover them and does not limit the rights that license gives you.

VelociQ8 is not affiliated with the developers of the original model, and this conversion is not endorsed by them.

License of this model

This repository contains a converted copy of intfloat/multilingual-e5-base. The weights were converted to Q8A64 and stored in a GGUF file. The model was not retrained or fine-tuned.

License: MIT License. The permission notice is in the LICENSE file of this repository. The original repository gives no copyright line and has no LICENSE file, so the text here has no copyright line either.

_{Arm is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere. Other product and hardware names belong to their respective owners and are used only to say what VelociQ8 is compared against or measured on.}