harrier-oss-v1-270m — OpenVINO IR for Intel NPU (AI Boost)

microsoft/harrier-oss-v1-270m (Gemma3, 640-dim, MIT) converted to OpenVINO IR so it runs as an embedding model on the Intel AI Boost NPU, and verified against the original PyTorch model on the NPU itself.

• Pooling is inside the graph. Inputs input_ids, attention_mask [B, L] (int64, right-padded) → output sentence_embedding [B, 640], last-token pooled and L2-normalized. Use it directly for cosine / dot product.
• The IR is saved with dynamic shapes. NPU requires static shapes, so reshape per sequence-length bucket at load time (the example uses [1, 2048] and [1, 8192]).

Quick start (NPU)

pip install openvino transformers huggingface_hub
hf download doldolggi/harrier-oss-v1-270m-OpenVINO --local-dir harrier-ov
python harrier-ov/run_npu.py            # FP16 on NPU
python harrier-ov/run_npu.py --int8     # INT8 on NPU  (INT4: --int4)

from run_npu import HarrierOV
model = HarrierOV("harrier-ov", device="NPU")          # int8=True for the INT8 variant
q = model.encode(["how much protein should a female eat"], query=True)   # adds the instruct prefix
d = model.encode(["As a general guideline, the CDC's average requirement of protein ..."])
print(q @ d.T)

Queries need the instruction prefix (documents do not), same as the original model: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: ". Other prompts are in config_sentence_transformers.json.

⚠️ First NPU compile is slow (measured: ~15 s for L=2048, ~3 min for L=8192). run_npu.py sets CACHE_DIR, so subsequent loads reuse the compiled blob. If you only need short inputs, drop the 8192 bucket.

⚠️ Inputs longer than 8192 tokens are truncated (EOS is kept — last-token pooling reads the EOS position).

Verification

Reference: original microsoft/harrier-oss-v1-270m in PyTorch FP32, last-token pooling + L2 normalize (as in the model card). Eight texts (Korean / English / Japanese, 6 → 8192 tokens, the last one truncated at 8192), padded to the static bucket. Numbers are cosine similarity vs. the reference.

Model-card example (run_npu.py, query × document scores ×100):

NPU compile: ~15 s (L=2048) + ~3 min (L=8192) the first time; 2.3 s with the CACHE_DIR cache warm.

Measured on Intel Core Ultra 7 270K Plus (NPU "Intel(R) AI Boost", 37xx), Ubuntu 24.04, kernel 7.0, linux-npu-driver 1.38.0, Level Zero 1.32.0, NPU firmware 20260820, OpenVINO 2026.4.0, NNCF 3.3.0. Latency is a single request at the padded bucket length (a short query still pays for 2048 tokens).

Conversion notes — why this is not a plain convert_model + compress_to_fp16

Two problems, both silent (no error, plausible-looking unit-norm vectors):

1. Gemma3's residual stream overflows FP16. Measured max activation 131,781 (> 65,504) from layer 7 on, concentrated at the BOS position. The NPU only computes in FP16 (INFERENCE_PRECISION_HINT=f32 is rejected), so a straight conversion gives cosine ≈ 0.98 on every input, even 6 tokens (FP16 and INT8 alike). Fix — exact reparametrization, residual stream scaled by 1/64 (done in PyTorch before export):

   • embed_tokens.embed_scale /= 64
   • post_attention_layernorm, post_feedforward_layernorm: (1 + w) → (1 + w) / 64
   • input_layernorm, pre_feedforward_layernorm, final norm: eps /= 64²

   RMSNorm is scale-invariant, so this is mathematically identical (FP32 check: worst cosine 0.99999994 vs the original). Max residual becomes 2,059. The norm weights therefore differ from the original checkpoint on purpose. The embedding weights are deliberately left untouched: scaling them instead makes INT8 per-channel scales so small (~1.7e-4) that INT8 on NPU collapses to cosine 0.24–0.39 (while the same file is fine on CPU).

2. save_model(compress_to_fp16=True) also rounds the RoPE inv_freq constant, which drifts with position: cosine 0.9925 at 8192 tokens on CPU. Neither the disable_fp16_compression rt_info nor an f64 constant prevented it, so weights are converted to FP16 explicitly and everything else is kept in FP32.

export.py and verify.py reproduce the conversion and the table above (they expect the original checkpoint in src_270m/: hf download microsoft/harrier-oss-v1-270m --local-dir src_270m).

INT4 — ported from Unsloth's UD recipe

INT4 is not a plain compress_weights(mode=INT4_SYM). Channel-wise data-free INT4 loses too much (worst cosine 0.933 on 270m, 0.941 on 0.6b, measured on NPU). What is published here instead takes the per-tensor bit allocation from Unsloth's Dynamic GGUF of the base model and ports it to OpenVINO:

1. Read the tensor types out of UD-Q4_K_XL (only the GGUF header is fetched, via an HTTP range request) — base models unsloth/gemma-3-270m-it-GGUF and unsloth/Qwen3-0.6B-GGUF.
2. Tensors that Unsloth keeps at 4 bit (Q4_K / IQ4_XS) → NNCF INT4_SYM, group size 32, with scale estimation (plus AWQ where it applies).
3. Everything Unsloth raises to 5/6/8 bit, and the embedding table → INT8_SYM.

The published row is the worst of the eight verification texts above, on NPU. The two alternatives were only screened — five short/medium texts on CPU — so their numbers are the optimistic end, not a like-for-like worst case:

ud_apply.py in this repo reproduces it; extract_ud_recipe.py (from the EvoCUA GGUF work) does step 1.

⚠️ On 270m, INT4 is nearly pointless — 250 MiB vs 257 MiB for INT8. Most of this model is the 262144×640 embedding table, which stays INT8, and Unsloth's recipe only puts 22 of 126 layer tensors at 4 bit. It is published for completeness; use INT8 unless you have a specific reason. AWQ could not be applied here (it needs the paired projection, which the recipe keeps at INT8, and raises KeyError), so only scale estimation was used.

License

MIT, same as the base model. All credit for the model goes to Microsoft.