bge-m3 GGUF (F16)

BAAI/bge-m3 dense embeddings as a GGUF for llama.cpp: 1024 dimensions, 8192-token context, 100+ languages.

Converted from the original FP32 checkpoint with llama.cpp b11703 convert_hf_to_gguf.py --outtype f16: a cast to F16, no quantization. Dense output only (CLS pooling); the sparse and ColBERT heads of bge-m3 are not included.

Run

llama-server -m bge-m3-f16.gguf --embedding --pooling cls -ngl 99 -c 8192 -b 8192 -ub 8192 --port 8080

curl -s localhost:8080/v1/embeddings -H 'Content-Type: application/json' -d '{"input": ["Hello world"]}'

Docker images (llama.cpp b11703):

docker run -d --gpus all -v $PWD:/models:ro -p 8080:8080 myllmbox/bge-m3-llamacpp:v1-nvidia \
  -m /models/bge-m3-f16.gguf --embedding --pooling cls -ngl 99 -c 8192 -b 8192 -ub 8192 --host 0.0.0.0 --port 8080
docker run -d --device /dev/dri -v $PWD:/models:ro -p 8080:8080 myllmbox/bge-m3-llamacpp:v1-amd \
  -m /models/bge-m3-f16.gguf --embedding --pooling cls -ngl 99 -c 8192 -b 8192 -ub 8192 --host 0.0.0.0 --port 8080

How to send

Keep about 4 requests in flight per server (llama-server's slots, -np, default 4), one page per request. Fewer leaves the GPU idle between requests; more only queues.

Speed

A4-like pages of ~565 tokens, the image above, -c 8192 -b 8192 -ub 8192:

Requests in flight × pages per request (pages/s):

Agreement with the original

18 texts (short queries, sentences, ~600-token pages; English, Lithuanian, German, French), cosine of each text's vector against the original FP32 model (transformers, CLS, normalised):

Vectors from any of these boxes can share one index.

License: MIT, as the original model.