Qwen3.5-4B

This version of Qwen3.5-4B has been converted to run on the Axera NPU using w4a16 quantization.

Compatible with Pulsar2 version: 5.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo :

https://huggingface.co/Qwen/Qwen3.5-4B

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU HOST LLM Runtime

Support Platform

AX650
- AX650N DEMO Board
- M4N-Dock(爱芯派Pro)
- M.2 Accelerator card

Image Process

Chips	input size	image num	ttft(168 tokens)	w4a16	CMM	Flash
AX650	384*384	1	845 ms	5.7 tokens/sec	4.91GiB	5.59GiB

Video Process

Chips	input size	image num	ttft(600 tokens)	w4a16	CMM	Flash
AX650	384*384	8	2111 ms	5.7 tokens/sec	4.91GiB	5.59GiB

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

How to use

安装 axllm

方式一：克隆仓库后执行安装脚本：

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

方式二：一行命令安装（默认分支 axllm）：

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）：

如果没有编译环境，请到： https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm 下载 最新 CI 导出的可执行程序（axllm），然后：

chmod +x axllm
sudo mv axllm /usr/bin/axllm

模型下载（Hugging Face）

mkdir -p AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K
cd AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K
hf download AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K --local-dir .

# structure of the downloaded files
tree -L 3
`-- AXERA-TECH
    `-- Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K
        |-- qwen3_5_vision.axmodel
        |-- README.md
        |-- config.json
        |-- image.png
        |-- model.embed_tokens.weight.bfloat16.bin
        |-- post_config.json
        |-- qwen3_5_tokenizer.txt
        |-- qwen3_5_text_p128_l0_together.axmodel
        ...
        |-- qwen3_5_text_p128_l23_together.axmodel
        |-- qwen3_5_text_post.axmodel
        `-- vision_cache

3 directories, 39 files

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

运行（CLI）

root@ax650 ~/yongqiang/lhj/Qwen3_5.AXERA/ax-llm # axllm run Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K/
11:40:21.412 INF Init:218 | LLM init start
11:40:21.412 INF Init:226 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
tokenizer_type = 3
 96% | ##############################   |  26 /  27 [3.78s<3.93s, 6.87 count/s] init post axmodel ok,remain_cmm(5589 MB)
11:40:25.195 INF Init:368 | max_token_len : 2047
11:40:25.195 INF Init:371 | kv_cache_size : 512, kv_cache_num: 2047
11:40:25.195 INF Init:374 | prefill_token_num : 128
11:40:25.195 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
11:40:25.195 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
11:40:25.195 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
11:40:25.195 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
11:40:25.195 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
11:40:25.195 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 768
11:40:25.195 INF Init:379 | grp: 7, prefill_max_kv_cache_num : 1024
11:40:25.195 INF Init:379 | grp: 8, prefill_max_kv_cache_num : 1152
11:40:25.195 INF Init:384 | prefill_max_token_num : 1152
11:40:25.195 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  27 /  27 [3.79s<3.79s, 7.13 count/s] embed_selector init ok
11:40:25.478 INF Init:643 | Qwen-VL token ids: vision_start=248053 image_pad=248056 video_pad=248057
11:40:25.478 INF Init:668 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
11:40:25.478 WRN Init:677 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
11:40:25.480 INF load_config:282 | load config: 
11:40:25.480 INF load_config:282 | {
11:40:25.480 INF load_config:282 |     "enable_repetition_penalty": false,
11:40:25.480 INF load_config:282 |     "enable_temperature": false,
11:40:25.480 INF load_config:282 |     "enable_top_k_sampling": true,
11:40:25.480 INF load_config:282 |     "enable_top_p_sampling": false,
11:40:25.480 INF load_config:282 |     "penalty_window": 20,
11:40:25.480 INF load_config:282 |     "repetition_penalty": 1.2,
11:40:25.480 INF load_config:282 |     "temperature": 0.9,
11:40:25.480 INF load_config:282 |     "top_k": 10,
11:40:25.480 INF load_config:282 |     "top_p": 0.8
11:40:25.480 INF load_config:282 | }
11:40:25.480 INF Init:448 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input image path (empty = text-only). Use "video:<frames_dir>" for video.
----------------------------------------
prompt >> 描述视频内容
media >> video:assets/football.mp4:1.5
17:54:19.424 INF extract_video_frames_ffmpeg:299 | Extracting raw video container to frames: assets/football.mp4 -> /tmp/axllm_video_frames/video_77305085315876_0 fps=1.4985014985014984
17:54:40.848 INF collect_video_frame_paths:379 | Video fps sampling: path=assets/football.mp4 fps=1.5 duration=60.060s target_frames=90 selected=90
17:55:01.670 INF SetKVCache:2543 | decode_grpid:3 prefill_grpid:6 history_cap:0 total_cap:256 symbolic_cap:1 precompute_len:0 input_num_token:6504 prefer_symbolic_group:0
17:55:01.670 INF SetKVCache:2565 | current prefill_max_token_num:10496
17:55:02.375 INF SetKVCache:2581 | first run
17:55:02.481 INF Run:2738 | input token num : 6504, prefill_split_num : 26
17:55:02.481 INF Run:2818 | prefill chunk p=0 history_len=0 grpid=6 kv_cache_num=0 input_tokens=256
17:55:03.405 INF Run:2818 | prefill chunk p=1 history_len=256 grpid=8 kv_cache_num=512 input_tokens=256
17:55:04.400 INF Run:2818 | prefill chunk p=2 history_len=512 grpid=9 kv_cache_num=768 input_tokens=256
17:55:05.385 INF Run:2818 | prefill chunk p=3 history_len=768 grpid=10 kv_cache_num=1024 input_tokens=256
17:55:06.362 INF Run:2818 | prefill chunk p=4 history_len=1024 grpid=11 kv_cache_num=1280 input_tokens=256
17:55:07.365 INF Run:2818 | prefill chunk p=5 history_len=1280 grpid=12 kv_cache_num=1536 input_tokens=256
17:55:08.399 INF Run:2818 | prefill chunk p=6 history_len=1536 grpid=13 kv_cache_num=1792 input_tokens=256
17:55:09.466 INF Run:2818 | prefill chunk p=7 history_len=1792 grpid=14 kv_cache_num=2048 input_tokens=256
17:55:10.592 INF Run:2818 | prefill chunk p=8 history_len=2048 grpid=15 kv_cache_num=2304 input_tokens=256
17:55:11.649 INF Run:2818 | prefill chunk p=9 history_len=2304 grpid=16 kv_cache_num=2560 input_tokens=256
17:55:12.724 INF Run:2818 | prefill chunk p=10 history_len=2560 grpid=17 kv_cache_num=2816 input_tokens=256
17:55:13.798 INF Run:2818 | prefill chunk p=11 history_len=2816 grpid=18 kv_cache_num=3072 input_tokens=256
17:55:14.886 INF Run:2818 | prefill chunk p=12 history_len=3072 grpid=19 kv_cache_num=3328 input_tokens=256
17:55:15.985 INF Run:2818 | prefill chunk p=13 history_len=3328 grpid=20 kv_cache_num=3584 input_tokens=256
17:55:17.091 INF Run:2818 | prefill chunk p=14 history_len=3584 grpid=21 kv_cache_num=3840 input_tokens=256
17:55:18.232 INF Run:2818 | prefill chunk p=15 history_len=3840 grpid=22 kv_cache_num=4096 input_tokens=256
17:55:19.386 INF Run:2818 | prefill chunk p=16 history_len=4096 grpid=23 kv_cache_num=4352 input_tokens=256
17:55:20.613 INF Run:2818 | prefill chunk p=17 history_len=4352 grpid=24 kv_cache_num=4608 input_tokens=256
17:55:21.853 INF Run:2818 | prefill chunk p=18 history_len=4608 grpid=25 kv_cache_num=4864 input_tokens=256
17:55:23.114 INF Run:2818 | prefill chunk p=19 history_len=4864 grpid=26 kv_cache_num=5120 input_tokens=256
17:55:24.395 INF Run:2818 | prefill chunk p=20 history_len=5120 grpid=27 kv_cache_num=5376 input_tokens=256
17:55:25.681 INF Run:2818 | prefill chunk p=21 history_len=5376 grpid=28 kv_cache_num=5632 input_tokens=256
17:55:26.995 INF Run:2818 | prefill chunk p=22 history_len=5632 grpid=29 kv_cache_num=5888 input_tokens=256
17:55:28.322 INF Run:2818 | prefill chunk p=23 history_len=5888 grpid=30 kv_cache_num=6144 input_tokens=256
17:55:29.667 INF Run:2818 | prefill chunk p=24 history_len=6144 grpid=31 kv_cache_num=6400 input_tokens=256
17:55:31.043 INF Run:2818 | prefill chunk p=25 history_len=6400 grpid=32 kv_cache_num=6656 input_tokens=104
17:55:32.459 INF Run:3045 | ttft: 29978.45 ms
<think>17:55:32.459 INF Run:3076 | VLM decode positions: rope_start=6372 dense_kv_start=6504


</think>

这段视频是一个关于足球比赛的直播或录像，具体是2019年1月30日巴塞罗那队与塞维利亚队的比赛。视频展示了比赛中的精彩瞬间和进球。

- **比赛场景**：视频展示了一场足球比赛的片段，场地为标准的足球场，有清晰的白色线条标记。
- **球员和队服**：
  - 巴塞罗那队穿着蓝色和红色条纹的球衣，号码为10的球员是路易斯·苏亚雷斯（Luis Suárez）。
  - 塞维利亚队穿着白色球衣。
- **比赛动作**：
  - 在多个片段中，可以看到球员们在场上积极跑动和传球。
  - 一个关键片段显示，巴塞罗那队在进攻中成功突破防守，最终由苏亚雷斯打入一球。
  - 进球后，苏亚雷斯与队友庆祝，场面充满激情。
- **观众和氛围**：
  - 观众席上坐满了观众，他们热情地为比赛加油。
  - 场边的广告牌上可以看到“Nike”和“Estrella Damm”等品牌标识。
- **直播者**：
  - 在视频的右上角，有一个直播者正在观看和评论比赛。他穿着巴塞罗那队的球衣，表情丰富，时而惊讶，时而兴奋，显然对比赛非常投入。
- **技术细节**：
  - 视频底部有“FBS00MPS10"的水印，以及播放进度条，显示视频正在播放中。
  - 视频标题为“Barcelona vs Sevilla 6-1 All Goals & Extended Highlights 30/01/2019 HD"，表明这是该场比赛的完整进球和扩展集锦。

这段视频通过多个镜头展示了比赛的紧张和精彩，以及球迷和直播者对比赛的热情。

17:56:46.246 NTC Run:3445 | hit eos,decode avg 5.24 token/s
17:56:46.246 INF GetKVCache:2496 | precompute_len:6892, remaining:3604 (tracked)

启动服务（OpenAI 兼容）

root@ax650:~# axllm serve AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K
[I][                            Init][ 138]: LLM init start
tokenizer_type = 1
 96% | ███████████████████████████████   |  30 /  31 [4.63s<4.79s, 6.47 count/s] init post axmodel ok,remain_cmm(9563 MB)
[I][                            Init][ 199]: max_token_len : 2047
[I][                            Init][ 202]: kv_cache_size : 1024, kv_cache_num: 2047
[I][                            Init][ 205]: prefill_token_num : 128
[I][                            Init][ 209]: grp: 1, prefill_max_kv_cache_num : 1
[I][                            Init][ 209]: grp: 2, prefill_max_kv_cache_num : 128
[I][                            Init][ 209]: grp: 3, prefill_max_kv_cache_num : 256
[I][                            Init][ 209]: grp: 4, prefill_max_kv_cache_num : 384
[I][                            Init][ 209]: grp: 5, prefill_max_kv_cache_num : 512
[I][                            Init][ 209]: grp: 6, prefill_max_kv_cache_num : 640
[I][                            Init][ 209]: grp: 7, prefill_max_kv_cache_num : 768
[I][                            Init][ 209]: grp: 8, prefill_max_kv_cache_num : 896
[I][                            Init][ 209]: grp: 9, prefill_max_kv_cache_num : 1024
[I][                            Init][ 209]: grp: 10, prefill_max_kv_cache_num : 1152
[I][                            Init][ 214]: prefill_max_token_num : 1152
[I][                            Init][  27]: LLaMaEmbedSelector use mmap
100% | ████████████████████████████████ |  31 /  31 [4.64s<4.64s, 6.69 count/s] embed_selector init ok
[W][                            Init][ 457]: Qwen-VL vision size override: cfg=448x448 bytes=1204224, model_input_bytes=884736 -> 384x384 (square).
[I][                            Init][ 641]: Qwen-VL token ids: vision_start=151652 image_pad=151655 video_pad=151656
[I][                            Init][ 666]: VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
[I][                            Init][ 672]: VisionModule deepstack enabled: layers=3
[I][                     load_config][ 282]: load config:
{
    "enable_repetition_penalty": false,
    "enable_temperature": false,
    "enable_top_k_sampling": false,
    "enable_top_p_sampling": false,
    "penalty_window": 20,
    "repetition_penalty": 1.2,
    "temperature": 0.9,
    "top_k": 10,
    "top_p": 0.8
}

[I][                            Init][ 272]: LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K'...
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K

OpenAI 调用示例

from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
completion = client.chat.completions.create(
    model=MODEL,
    messages=messages,
)

print(completion.choices[0].message.content)

OpenAI 流式调用示例

from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C256-P10K-CTX12K"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
stream = client.chat.completions.create(
    model=MODEL,
    messages=messages,
    stream=True,
)

print("assistant:")
for ev in stream:
    delta = getattr(ev.choices[0], "delta", None)
    if delta and getattr(delta, "content", None):
        print(delta.content, end="", flush=True)
print("
")

AXERA-TECH/Qwen3.5-4B-AX650-GPTQ-Int4-C512-P30k-CTX32k