🇰🇷 한국어 카드: MODEL_CARD.ko.md

Qwen3-Embedding-0.6B-MultiCal-M1a-GPTQ (Research Preview)

A low-bit copy of Qwen3-Embedding-0.6B, quantized with a GPTQ-style method to a mixed 3-bit/4-bit configuration and shipped here as a GGUF file that runs on llama.cpp. The GGUF is a value-preserving re-encoding of our internal packed artifact: every reconstructed weight is bit-identical, so you can run the model with a standard loader and no code from us.

Use it if you work on low-bit multilingual retrieval embeddings and want a quality reference point you can actually execute. It is a research preview, not a tuned serving artifact.

What is in this repository

The packed .safetensors is not loadable with AutoModel.from_pretrained() or SentenceTransformer(...), and this repository ships no loader for it. Its per-tensor bit width and group size live in the safetensors header under __metadata__["mcq"]; the layout is row-major flat and the decode rule is value = code * scale + min with fp16 scale and min per group. That is enough to write your own dequantizer, but if you just want to run the model, use the GGUF.

How to run it

The GGUF carries pooling_type = LAST and add_eos_token = true, so llama.cpp reproduces our pooling without any flags. The raw output vector is not normalized — L2-normalize it before computing cosine similarity, exactly as with the base model.

# llama.cpp, built from source (CPU build is enough for a 0.6B model)
./llama-embedding -m model-q4_1.gguf --pooling last -p "The capital of France is Paris." \
    --threads 8 --threads-batch 8 --embd-normalize 2

# llama-cpp-python
import numpy as np, llama_cpp

llm = llama_cpp.Llama(
    model_path="model-q4_1.gguf",
    embedding=True,
    n_ctx=512, n_batch=512, n_ubatch=512,
    n_threads=8,
    n_threads_batch=8,   # see "CPU threads" below — do not leave this unset in a container
)

def embed(text):
    v = np.asarray(llm.embed(text), dtype=np.float32).reshape(-1)   # 1024 dims, unnormalized
    return v / np.linalg.norm(v)

docs = ["한국의 수도는 서울이다.", "The capital of France is Paris."]
q = embed("대한민국의 수도는 어디인가?")
print(np.stack([embed(d) for d in docs]) @ q)

Follow the base model's card for query instructions and prompt formats; this release does not change them.

CPU threads — one setting that matters a lot

n_threads alone is not enough. llama.cpp uses a separate thread count for the prompt-eval (batch) path, and when it is unset the library sizes its pool from the host core count. In our measurement container os.cpu_count() reported 256 cores while the cgroup quota was 8 (cpu.max = 800000 100000), and a single embedding call took 298-353 s. Adding n_threads_batch=8 brought the same calls to 0.04-0.07 s on the same file, same build and same machine.

That is a thread-oversubscription fix, not a model improvement, and the numbers are specific to that container. Treat 8 as "match your actual available cores", not as a recommended constant. Reducing n_ubatch did not help (346-354 s), so batch padding was not the cause.

Sizes — three numbers, kept separate

The GGUF is larger than the packed file, by design: Q4_1 stores every tensor at 4 bits with an fp16 scale and an fp16 min per 32 values, so the 3-bit groups that make the packed artifact small are padded out when they are written into a single-type GGUF container.

⚠️ Our internal comparison found the packed M1a layout about 0.95% smaller than a uniform 3-bit g32 baseline by payload accounting. That size result belongs to the packed format only and does not carry over to this GGUF. Do not cite the GGUF as a 3-bit size win; it is a 4-bit file whose reconstructed values came from a mixed 3/4-bit quantization.

Conversion verification (what we checked on the GGUF)

Measured on the exact file published here.

The cosine comparison is against our own fp32 reference built from the same packed weights, so it measures the conversion, not retrieval quality. The top-1 check is a 9-sentence smoke test: it shows the vectors are usable, and it is far too small to be read as a retrieval score.

We did not re-run the retrieval benchmarks through llama.cpp. The quality table below was measured on the packed artifact with our internal fp32 loader.

Retrieval quality of the original packed model

nDCG@10, mean of three build seeds, against a uniform 3-bit g32 baseline built from the same calibration data. Verdicts follow a preregistered rule combining a seed-noise floor with a paired bootstrap interval. These numbers describe the packed artifact, not the GGUF.

Development sets:

Sealed sets, used once for this comparison:

One internal Korean development set informs our decisions and is not reported here. "Not attributable" means the difference sits inside build noise. It does not mean the two are equal.

Configuration of the quantization

Reproducibility evidence

All three seeds share the layout and size (packed payload 295,029,952 B, weight file 295,391,600 B). Only seed 0 is published; the others are listed so the seed spread behind the quality table can be checked.

The spread across seeds is of the same order as the differences in the quality table, which is why that table reports noise-floor verdicts rather than raw gaps. Reloading seed 0 from the published packed file alone and re-scoring NFCorpus on a different device (Apple MPS, fp32) gave 33.03 against 32.97 in our ledger (H100), a difference of 0.05.

Limitations

• Research preview. No latency, throughput or peak-memory numbers are published for this model. The thread finding above is an operational note, not a benchmark.
• The packed INT3 format has no direct execution path here. There is no low-bit kernel for it and no loader in this repository. The runnable path is the Q4_1 GGUF, where llama.cpp does perform quantized inference; it is a 4-bit container, so do not describe it as 3-bit serving.
• Quality was measured on the packed artifact, conversion on the GGUF. The two are linked by a 9-sentence cosine comparison (min 0.999020), not by a retrieval re-run.
• The top-1 figure is a smoke test over 6 documents and 3 queries.
• llama.cpp does not apply Unicode NFC normalization. The tokenizer configuration specifies NFC, so if your input is not already NFC the token sequence can differ from the HF tokenizer. Normalize your text to NFC before embedding.
• The cause of the quality gain is not attributed to early-layer protection. M1a changed the bit width and group size of layers 0-6 and the group size of the embedding table at the same time, and a follow-up test meant to localize the effect did not pass its preregistered gate. Read the result as the effect of the whole configuration.
• One model, one budget. Nothing here generalizes to other embedding models or byte budgets.
• No equivalence claim and no full-precision comparison. "No attributable loss" is weaker than "same quality", and the development sets are not hold-out sets.

Citation and credit

Base model by the Qwen team (Apache-2.0). Quantization, conversion and evaluation by ThakiCloud. Calibration corpora: FineWeb-Edu and FineWeb-2 (ODC-BY); neither the text nor the token ids are redistributed here.