Qwen3-0.6B compiled for the Synaptics
Astra™ SL2610-Series Torq NPU. The weights are INT4 and calibrated with GPTQ
on HuggingFaceH4/ultrachat_200k.
<think>\n\n</think>\n\n block, as apply_chat_template(..., enable_thinking=False) does.| File | Description |
|---|---|
transformer.vmfb |
Static single-token decoder. Inputs: token_embedding [1,1,1024] bf16, position_ids [1,1] i32, 28 × read-only time-major KV cache past_key_values.N.key_value [1,256,16,128] bf16 (K heads 0–7, V heads 8–15). Outputs: hidden state [1,1,1024] and 28 × present.N.key_value [1,1,16,128] bf16, the new token's K/V rows. The runtime writes these rows into the caches at position_ids before the next step. |
lm_head.vmfb.trim |
Vocabulary projection: hidden state → logits [1,1,99520] |
token_embeddings.npy |
bf16 [151936, 1024] embedding table. The lookup runs on the host. |
token_id_lut.npy |
int32 [99520]: maps a compact logit index to a tokenizer ID |
tokenizer.json, tokenizer_config.json, config.json, generation_config.json |
From Qwen/Qwen3-0.6B |
gptq/ |
GPTQ INT4 weights that torq-export-model qwen uses |
Built with torq-tools:
torq-export-model qwen -s qwen3-0.6b
repeat_kv
copy.DequantizeLinear(block_size=32) with bf16 scales and INT4 zero points.
The weights only are quantized; activations stay bf16. GPTQ
(llmcompressor, asymmetric, group size 32) calibrated the weights on 256
samples × 1024 tokens.--torq-dma-in-mtu=6 --torq-dma-out-mtu=6). The compile flags are in torq-tools
src/torq/models/qwen/compile_flags.json.The read-only KV cache needs the torq-examples qwen3 demo with read-only KV
cache support (torq-examples #60 or later).
The weight-quantization error was measured against the fp32 static model, with teacher-forced logits at 288 positions from 6 chat prompts:
| KL divergence | Cosine | Top-1 agreement | |
|---|---|---|---|
| GPTQ INT4 (this model) | 0.107 | 0.968 | 85.4 % |
| RTN INT4 (reference) | 0.152 | 0.948 | 83.3 % |
These values were measured on an SL2610 board with the NPU at 1 GHz and the
torq-examples qwen3 demo:
| Time per call | |
|---|---|
transformer.vmfb |
93.6 ms |
lm_head.vmfb.trim |
14.7 ms |
| Decode (chat demo, non-thinking) | 8.7 tok/s |
| Time to first token (short question after the system prompt) | 1.9–2.1 s |
Qwen3-0.6B is released by the Qwen team under the Apache License 2.0. See NOTICE.txt.
Qwen/Qwen3-0.6B