Qwen3-Embedding-0.6B for NInfer Extended

A native BF16 conversion of Qwen3-Embedding-0.6B for text embeddings, semantic retrieval and similarity. All 310 source BF16 parameters are preserved, and the tokenizer is embedded in the artifact. Tested on RTX 5070 Ti and RTX 6000D.

Build and run

Use NInfer Extended from the main branch to run this artifact.

git clone https://github.com/ByronLeeeee/ninfer-extended.git
cd ninfer-extended
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCUDNN_ROOT=/path/to/cudnn -DCUBLAS_ROOT=/path/to/cublas
cmake --build build --target ninfer-embed -j
pip install "transformers==5.17.0" numpy
pip install huggingface_hub
hf download ByronLeeee/Qwen3-Embedding-0.6B-Ninfer qwen3-embedding-0.6b-bf16.ninfer --local-dir models

Build dependencies: 64-bit Linux (WSL2 on Windows), CUDA supporting sm_120a, C++20, CMake ≥3.28, Ninja, FFmpeg development libraries, libcurl, PCRE2, cuDNN 9 and cuBLAS. Omit CUBLAS_ROOT when CUDA provides cuBLAS.

Save the input as texts.json:

{"texts": ["A passage for semantic retrieval.", "Text embeddings support semantic search."]}

python tools/qwen3_embedding/embed.py \
  --artifact models/qwen3-embedding-0.6b-bf16.ninfer \
  --binary build/apps/ninfer-embed --input texts.json --out vectors.json \
  --dimensions 1024

The output contains normalized 1,024-dimensional vectors. For retrieval queries, add --instruction "Given a web search query, retrieve relevant passages that answer the query"; documents use their raw text.

The frontend batches up to eight texts and 32,768 total unpadded tokens at a time. Larger collections are split into batches. Supported output dimensions are 32, 128, 256, 512 and 1024.

Conversion guide and Engine API

Transformers vs NInfer performance

Both engines use BF16 and produce normalized 1,024-dimensional vectors. GPU timings include the complete language prefill and vector pooling on pre-tokenized input; weight loading, tokenization and initial graph capture are excluded. Text embedding has no decode phase.

Each Transformers baseline uses the faster eager or fullgraph-compiled path. All 310 parameter tensors and computation modules were checked on CUDA, with fast CUDA attention and MATH SDPA disabled. Compiled execution has zero graph breaks and error suppression is disabled. Environment: PyTorch 2.14.0+cu132 and Transformers 5.17.0; CUDA toolkit 13.3 on 5070 Ti and 13.2 on 6000D.

Changes use Transformers as the denominator: (TF time / NInfer time − 1) × 100%. The two engines are effectively tied for 2,048 tokens on 6000D. Values are medians: 30 measured NInfer forwards per case, or 62 for local long input. The audited baseline and NInfer runs use the same GPU and input settings in separate windows; the 6000D 2,048-token reference was refreshed with 31 measured forwards.

Vector quality and result agreement

Both eager and compiled Transformers outputs are checked. Vector agreement covers 249 vectors; semantic similarity uses the first 100 STS-B pairs; the Chinese/English retrieval set contains eight queries.

Both engines retrieve the correct Top1 document for all eight queries, with identical Top1 and Top3 document sets. Floating-point vectors differ slightly, with mean cosine around 0.9998 and closely matching Spearman scores. Each GPU passes 164 independent FP64 mathematical checks.

Memory and artifact

Tracked GPU weight storage is about 1.110 GiB, with no KV cache. Explicit runtime buffers use about 15.04 MiB for 4×128, 120.14 MiB for 8×512, and 0.938 GiB at 32K total capacity; CUDA driver and Graph objects add overhead.

qwen3-embedding-0.6b-bf16.ninfer · 1,203,049,125 bytes · BF16 · Apache-2.0

SHA256: 3229813fdae278ab6871f712e8a943fc4c8949d5580567e89f27f2f18d4e6ae6

Measurements · Artifact manifest · Conversion record

Qwen3-Embedding · STS-B

Current-main regression check

Inputs cover 1–8,192 tokens, batches of 1/4/8 and ragged lengths. All 258 vector positions are bit-identical between builds and eager execution; retrieval remains 8/8 correct, and Pearson correlation on 100 STS-B pairs is 0.92563. No confirmed performance regression.