all-MiniLM-L6-v2 GGUF (F32)

Model Description

• Sentence-BERT embedding model, converted from sentence-transformers/all-MiniLM-L6-v2
• Architecture: BERT (BertModel), 6 layers, 384-dim embeddings
• Max context length: 512 tokens
• Format: GGUF F32 (full precision, ~90 MB)

Conversion

Converted using llama.cpp convert_hf_to_gguf.py:

python convert_hf_to_gguf.py --outfile all-MiniLM-L6-v2-f32.gguf --outtype f32 ./all-MiniLM-L6-v2

Usage with llama.cpp

llama-server \
  --model all-MiniLM-L6-v2-f32.gguf \
  --embeddings \
  --host 0.0.0.0 --port 8080 \
  -c 512 \
  -fa on

Then call POST /v1/embeddings:

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "hello world"}'

Response (384-dim vector):

{
  "object": "list",
  "data": [{"object": "embedding", "embedding": [...], "index": 0}],
  "model": "all-MiniLM-L6-v2-f32.gguf",
  "usage": {"prompt_tokens": 5, "total_tokens": 5}
}

Python SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.embeddings.create(
    model="all-MiniLM-L6-v2-f32.gguf",
    input="hello world"
)
print(len(resp.data[0].embedding))  # 384

Docker Deployment

Ready-to-use Docker setup with multi-stage CUDA build, auto model download, and benchmark scripts:

llm-docker/models/all-minilm-l6-v2-f32

cd models/all-minilm-l6-v2-f32
cp .env.example .env
./download-model.sh
docker compose up -d --build

Files

• all-MiniLM-L6-v2-f32.gguf — F32 GGUF model (~90 MB)