MiniCPM5-2B for AMD XDNA 2 NPU (FastFlowLM)

This repository contains the quantized AMD Q4NX weights, AIE-ML firmware kernels, and runtime configuration for running openbmb/MiniCPM5-2B natively on the AMD XDNA 2 NPU (/dev/accel/accel0, 48 AIE-ML tiles) using FastFlowLM.

📦 Porting Pipeline & Diagnostic Harnesses: The full conversion scripts, architectural adaptation code, and 42-layer ERT timeout reproduction suite are hosted on GitHub at julianmb/minicpm5-xdna2.

---

Measured Performance Benchmarks

Directly measured on an AMD Strix Halo (Ryzen AI Max+ 395) workstation with FastFlowLM v1.0.2:

⚠️ Attribution caveat: these generation figures predate served-model verification. Per ROCm/FastFlowLM#716, an unresolvable tag can silently execute the resident model while echoing the requested name, and our harnesses did not check the response model field or /v1/models. Treat the decoding/prefill numbers below as unverified for MiniCPM5 until re-measured with served-model assertions (now added to the repo harnesses).

---

Hardware Support Matrix

FastFlowLM targets the AMD XDNA 2 NPU architecture found across the Ryzen AI 300 and Ryzen AI 400 families.

[!IMPORTANT] Testing Disclaimer: While the compiled AIE kernels and FastFlowLM runtime target all XDNA 2 processors uniformly, this port was specifically benchmarked, tuned, and verified on the AMD Strix Halo (Ryzen AI Max+ 395 w/ Radeon 8060S, 128 GB UMA). Community feedback and verification reports on Strix Point and Krackan Point laptops are warmly welcomed.

---

Technical Deep Dive: What We Did & How It Works

MiniCPM5-2B features 42 transformer layers with $d_{model} = 2048$, $d_{ffn} = 6144$, and $d_{head} = 128$. However, deploying an unmodified MiniCPM5-2B checkpoint onto FastFlowLM encounters two fatal hardware/firmware incompatibilities:

1. The GQA Ratio Constraint in AIE Firmware

MiniCPM5-2B is configured with 16 Query heads and 2 Key/Value heads ($16:2 = 8:1$ GQA ratio). Disassembly of FastFlowLM's multi-head attention firmware library (libmha.so) revealed that AMD's compiled AIE kernels only support:

• _gen_mha_seq_d64_q4 ($d_{head}=64$, $4:1$ ratio)
• _gen_mha_seq_d128_q2 ($d_{head}=128$, $2:1$ ratio)
• _gen_mha_seq_d128_q3 ($d_{head}=128$, $3:1$ ratio)
• _gen_mha_seq_d128_q4 ($d_{head}=128$, $4:1$ ratio)

There is no native $8:1$ AIE kernel for $d_{head}=128$. Additionally, FastFlowLM's Llama engine (libllama_npu.so) hardcodes hidden_size == 2048 to $d_{head}=64$.

2. The Solution: $4\times$ Mathematical KV Replication ($2 \to 8$ heads)

Under Grouped Query Attention:

• Query heads ${0..7}$ attend to Key/Value head $0$.
• Query heads ${8..15}$ attend to Key/Value head $1$.

By replicating Key/Value head $0$ four times into indices $[0, 1, 2, 3]$ and Key/Value head $1$ four times into indices $[4, 5, 6, 7]$ along axis 0:

• Query heads ${0, 1}$ attend to head $0$; ${2, 3}$ attend to head $1$; ${4, 5}$ attend to head $2$, etc.
• Because the copied heads contain identical weight values, the attention logits, softmax distributions, and output representations are bit-for-bit mathematically identical.
• The effective head count becomes $H_q = 16, H_{kv} = 8$ ($16:8 = 2:1$ ratio), perfectly matching the native _gen_mha_seq_d128_q2 AIE hardware kernel!

3. Qwen3 Engine Routing & QK-Norm Loader Shim (Not an Identity)

FastFlowLM's Qwen3 execution engine (libqwen3_npu.so) dynamically reads head_dim: 128 and selects the _gen_mha_seq_d128_q2 kernel for models with intermediate_size == 6144. To route through this engine:

1. The model is registered under family "qwen3" in model_list.json.
2. The Qwen3 engine requires model.layers.{i}.self_attn.q_norm.weight and k_norm.weight ($[128]$ BF16). Because MiniCPM5-2B has no QK-normalization, we injected synthetic unit tensors ($\gamma = 1.0$) into model.q4nx across all 42 layers. In RMSNorm: $$\text{RMSNorm}(x, \gamma = 1.0) = \frac{x}{\sqrt{\frac{1}{d}\sum x_i^2 + \epsilon}} \cdot 1.0$$ This satisfies the weight loader but does not preserve the original computation: unit-scale RMSNorm still normalizes Q and K, so attention scores differ from the un-normalized MiniCPM5 architecture. Output equivalence of this port is unvalidated.

---

Installation & Serving Guide

Linux (Ubuntu 24.04 / Linux 6.11+)

1. Clone the Model Repository

Clone directly into your user FastFlowLM models directory:

mkdir -p ~/.config/flm/models
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2 ~/.config/flm/models/MiniCPM5-2B-NPU2

2. Copy Kernel XCLBINs

FastFlowLM requires compiled AIE kernels to reside in its xclbins/ directory. Copy or symlink them:

# Locate your FLM installation root (e.g., ~/.config/flm or /var/lib/lemonade/.cache/lemonade/bin/flm/npu)
FLM_ROOT="${HOME}/.config/flm"
mkdir -p "${FLM_ROOT}/xclbins/MiniCPM5-2B-NPU2"
cp ~/.config/flm/models/MiniCPM5-2B-NPU2/*.xclbin "${FLM_ROOT}/xclbins/MiniCPM5-2B-NPU2/"

3. Register Model in FastFlowLM

Add the model definition to your FastFlowLM model_list.json (located in ~/.config/flm/model_list.json or /opt/flm/model_list.json):

{
  "models": {
    "minicpm5": {
      "2b": {
        "name": "MiniCPM5-2B-NPU2",
        "url": "https://huggingface.co/julianmb/MiniCPM5-2B-NPU2",
        "size": 2023518368,
        "flm_min_version": "0.9.22",
        "files": [
          "config.json",
          "model.q4nx",
          "tokenizer.json",
          "tokenizer_config.json"
        ],
        "default_context_length": 32768,
        "max_prefill_len": 4096,
        "details": {
          "family": "qwen3",
          "think": false,
          "parameter_size": "2B",
          "quantization_level": "Q4_1"
        },
        "label": ["general"],
        "footprint": 1.9
      }
    }
  }
}

4. Start the Server

flm serve minicpm5:2b --host 127.0.0.1 --port 8001

5. Test Inference via cURL

curl -s http://127.0.0.1:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "minicpm5:2b",
    "messages": [
      {"role": "user", "content": "Explain quantum superposition in simple terms with an analogy."}
    ],
    "max_tokens": 150,
    "temperature": 0.2
  }'

---

Windows 11 (AMD Ryzen AI / Strix Halo & Strix Point)

1. Prerequisites

1. Ensure the AMD NPU Driver is installed (part of AMD Ryzen AI Software / AMD Adrenalin).
   • Verify in Task Manager $\to$ Performance: You should see an NPU graph (device named AMD IPU Device or AMD NPU Device).
2. Install FastFlowLM for Windows using the official MSI (flm-setup.msi).

2. Download Model Weights

Open PowerShell and clone or copy the repository into your user profile directory:

New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\flm\models"
cd "$env:USERPROFILE\.config\flm\models"
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2

3. Copy Kernel XCLBINs

Copy the precompiled .xclbin files to your FastFlowLM xclbins directory:

$FLM_DIR = "$env:USERPROFILE\.config\flm"
New-Item -ItemType Directory -Force -Path "$FLM_DIR\xclbins\MiniCPM5-2B-NPU2"
Copy-Item "$FLM_DIR\models\MiniCPM5-2B-NPU2\*.xclbin" -Destination "$FLM_DIR\xclbins\MiniCPM5-2B-NPU2\"

4. Register the Model

Open C:\Program Files\FastFlowLM\model_list.json (or %USERPROFILE%\.config\flm\model_list.json) in a text editor (e.g., VS Code or Notepad with administrator privileges) and add the minicpm5 block under "models":

"minicpm5": {
  "2b": {
    "name": "MiniCPM5-2B-NPU2",
    "url": "https://huggingface.co/julianmb/MiniCPM5-2B-NPU2",
    "size": 2023518368,
    "flm_min_version": "0.9.22",
    "files": [
      "config.json",
      "model.q4nx",
      "tokenizer.json",
      "tokenizer_config.json"
    ],
    "default_context_length": 32768,
    "max_prefill_len": 4096,
    "details": {
      "family": "qwen3",
      "think": false,
      "parameter_size": "2B",
      "quantization_level": "Q4_1"
    },
    "label": ["general"],
    "footprint": 1.9
  }
}

5. Launch the Server via PowerShell

flm serve minicpm5:2b --host 127.0.0.1 --port 8001

6. Test Inference via PowerShell

$body = @{
    model = "minicpm5:2b"
    messages = @(
        @{ role = "user"; content = "Write a Python function to compute Fibonacci with memoization." }
    )
    max_tokens = 200
    temperature = 0.1
} | ConvertTo-Json

Invoke-RestMethod -Uri "http://127.0.0.1:8001/v1/chat/completions" `
    -Method Post `
    -Headers @{ "Content-Type" = "application/json" } `
    -Body $body | Select-Object -ExpandProperty choices | Select-Object -ExpandProperty message

---

Python OpenAI Client Example (Cross-Platform)

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8001/v1",
    api_key="none"  # FastFlowLM local server does not require an auth key
)

response = client.chat.completions.create(
    model="minicpm5:2b",
    messages=[
        {"role": "system", "content": "You are a helpful and concise assistant."},
        {"role": "user", "content": "What are the three laws of robotics?"}
    ],
    max_tokens=200,
    temperature=0.3
)

print(response.choices[0].message.content)

---

Repository Contents

MiniCPM5-2B-NPU2/
├── .gitattributes          # Git-LFS rules for *.q4nx, *.xclbin, tokenizer.json
├── README.md               # Model card & documentation
├── model.q4nx              # 1.88 GB AMD Q4NX weight tensor file
├── config.json             # FastFlowLM NPU SRAM addressing & runtime configuration
├── attn.xclbin             # Precompiled XDNA 2 Multi-Head Attention kernel (d128_q2)
├── mm.xclbin               # Precompiled XDNA 2 Matrix Multiply / SwiGLU kernel
├── layer.xclbin            # Precompiled XDNA 2 Transformer pipeline kernel
├── dequant.xclbin          # Precompiled XDNA 2 Q4_1 dequantization kernel
├── tokenizer.json          # Fast tokenizer vocabulary
├── tokenizer_config.json   # Tokenizer parameters
├── special_tokens_map.json # Special token IDs (BOS, EOS, PAD)
└── chat_template.jinja     # Jinja2 chat template formatting

---

Troubleshooting & FAQ

Q1: [ERROR] Failed to load model: [json.exception.type_error.302] type must be number, but is null

• Cause: FastFlowLM's C++ loader requires explicit integer token IDs in tokenizer_config.json.
• Fix: Fixed in the latest repository commit. Run git pull in your MiniCPM5-2B-NPU2 directory, or verify that your tokenizer_config.json contains:

  "bos_token_id": 0,
  "eos_token_id": [1, 130073],
  "pad_token_id": 1
  

  Note: eos_token_id must be a JSON array [1, 130073] (not a single scalar integer), where 130073 is <|im_end|>.

---

Q2: No such file '.../xclbins/MiniCPM5-2B-NPU2/layer.xclbin'

• Cause: FastFlowLM expects compiled .xclbin kernels inside <flm-root>/xclbins/MiniCPM5-2B-NPU2/.
• Fix: Copy all *.xclbin files from the cloned model folder into your FastFlowLM xclbins/MiniCPM5-2B-NPU2/ directory as described in Step 2 of the guide.

---

Q3: runlist failed execution (ERT_CMD_STATE_TIMEOUT) during Decode

• Symptoms: Prompt prefill succeeds (Prefill chunk 1/1 with ... tokens), but the server immediately returns {"error":"runlist failed execution (ERT_CMD_STATE_TIMEOUT)"} upon entering the autoregressive token generation phase (Start generating...).
• Technical Root Cause: In FastFlowLM's causal LM decode implementation (libqwen3_npu.so), generation dispatches through xrt::runlist (execute() → wait(), per disassembly). One hypothesis under investigation is that all transformer layer forward passes are chained into a single synchronous submission:

  call 24510 <xrt::runlist::execute()@plt>
  call 24710 <xrt::runlist::wait(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) const@plt>
  

  MiniCPM5-2B features 42 transformer layers, whereas the borrowed layer.xclbin and execution graph were compiled for 28 layers (Qwen3-1.7B-NPU2). It is hypothesized — but not yet demonstrated — that submitting 42 sequential layer executions in one chained xrt::runlist may exceed a command-queue depth or timeout threshold. A separate architectural-compatibility question is also open (see §3): the port re-labels a Llama-family model as Qwen3 and injects unit QK-norm scales that still normalize Q/K, so numerical equivalence is unvalidated independent of the dispatch failure.
• Correction regarding -noert: Earlier notes suggested building XRT with -noert. As confirmed in community testing, [-noert] in XRT's build.sh is strictly a build-time flag for building XRT without bundling Alveo PCIe ERT firmware (and is already implied by [-npu]); it is not a runtime scheduler switch and does not bypass the hardware command queue timeout.
• Engine-Specific Runlist Depth Boundary (37–42 Layers on qwen3): The observed pattern is consistent with an engine-specific limitation inside libqwen3_npu.so rather than a platform ceiling, but no controlled depth experiment (varying layer count while holding weights, kernels, and runtime fixed) has been run — treat the boundary as a hypothesis:
  • qwen3:1.7b (28 layers) decodes at 43.3 tok/s (qwen3 engine).
  • qwen3:4b (36 layers) decodes at 19.8 tok/s (qwen3 engine).
  • qwen3.6-moe:35b-a3b (40 layers) decodes at 16.8 tok/s (qwen3.6-moe engine).
  • gemma4-it:e4b (42 layers) decodes at 12.7 tok/s (gemma4e engine).
  • minicpm5:2b (42 layers) trips the runlist timeout on qwen3 engine. A queue-depth threshold between 37 and 42 layers for the qwen3 engine is suspected but unproven.
• Upstream Triage & Reproducer: A minimal standalone 10-second reproducer is available in julianmb/minicpm5-xdna2 (scripts/reproduce_ert_timeout.py), and the issue is actively tracked upstream in ROCm/FastFlowLM#712. Chunking the decode forward pass in libqwen3_npu.so (e.g. into two 21 + 21 layer sub-runlists) is an untested experimental proposal, not a demonstrated fix.l: openbmb/MiniCPM5-2B language:

• en
• zh license: apache-2.0 tags:
• fastflowlm
• xdna2
• npu
• amd
• strix-halo
• strix-point
• krackan
• ryzen-ai
• q4nx
• rocm pipeline_tag: text-generation

---

MiniCPM5-2B for AMD XDNA 2 NPU (FastFlowLM)

This repository contains the quantized AMD Q4NX weights, AIE-ML firmware kernels, and runtime configuration for running openbmb/MiniCPM5-2B natively on the AMD XDNA 2 NPU (/dev/accel/accel0, 48 AIE-ML tiles) using FastFlowLM.

📦 Porting Pipeline & Diagnostic Harnesses: The full conversion scripts, architectural adaptation code, and 42-layer ERT timeout reproduction suite are hosted on GitHub at julianmb/minicpm5-xdna2.

---

Measured Performance Benchmarks

Directly measured on an AMD Strix Halo (Ryzen AI Max+ 395) workstation with FastFlowLM v1.0.2:

⚠️ Attribution caveat: these generation figures predate served-model verification. Per ROCm/FastFlowLM#716, an unresolvable tag can silently execute the resident model while echoing the requested name, and our harnesses did not check the response model field or /v1/models. Treat the decoding/prefill numbers below as unverified for MiniCPM5 until re-measured with served-model assertions (now added to the repo harnesses).

---

Hardware Support Matrix

FastFlowLM targets the AMD XDNA 2 NPU architecture found across the Ryzen AI 300 and Ryzen AI 400 families.

[!IMPORTANT] Testing Disclaimer: While the compiled AIE kernels and FastFlowLM runtime target all XDNA 2 processors uniformly, this port was specifically benchmarked, tuned, and verified on the AMD Strix Halo (Ryzen AI Max+ 395 w/ Radeon 8060S, 128 GB UMA). Community feedback and verification reports on Strix Point and Krackan Point laptops are warmly welcomed.

---

Technical Deep Dive: What We Did & How It Works

MiniCPM5-2B features 42 transformer layers with $d_{model} = 2048$, $d_{ffn} = 6144$, and $d_{head} = 128$. However, deploying an unmodified MiniCPM5-2B checkpoint onto FastFlowLM encounters two fatal hardware/firmware incompatibilities:

1. The GQA Ratio Constraint in AIE Firmware

MiniCPM5-2B is configured with 16 Query heads and 2 Key/Value heads ($16:2 = 8:1$ GQA ratio). Disassembly of FastFlowLM's multi-head attention firmware library (libmha.so) revealed that AMD's compiled AIE kernels only support:

• _gen_mha_seq_d64_q4 ($d_{head}=64$, $4:1$ ratio)
• _gen_mha_seq_d128_q2 ($d_{head}=128$, $2:1$ ratio)
• _gen_mha_seq_d128_q3 ($d_{head}=128$, $3:1$ ratio)
• _gen_mha_seq_d128_q4 ($d_{head}=128$, $4:1$ ratio)

There is no native $8:1$ AIE kernel for $d_{head}=128$. Additionally, FastFlowLM's Llama engine (libllama_npu.so) hardcodes hidden_size == 2048 to $d_{head}=64$.

2. The Solution: $4\times$ Mathematical KV Replication ($2 \to 8$ heads)

Under Grouped Query Attention:

• Query heads ${0..7}$ attend to Key/Value head $0$.
• Query heads ${8..15}$ attend to Key/Value head $1$.

By replicating Key/Value head $0$ four times into indices $[0, 1, 2, 3]$ and Key/Value head $1$ four times into indices $[4, 5, 6, 7]$ along axis 0:

• Query heads ${0, 1}$ attend to head $0$; ${2, 3}$ attend to head $1$; ${4, 5}$ attend to head $2$, etc.
• Because the copied heads contain identical weight values, the attention logits, softmax distributions, and output representations are bit-for-bit mathematically identical.
• The effective head count becomes $H_q = 16, H_{kv} = 8$ ($16:8 = 2:1$ ratio), perfectly matching the native _gen_mha_seq_d128_q2 AIE hardware kernel!

3. Qwen3 Engine Routing & QK-Norm Loader Shim (Not an Identity)

FastFlowLM's Qwen3 execution engine (libqwen3_npu.so) dynamically reads head_dim: 128 and selects the _gen_mha_seq_d128_q2 kernel for models with intermediate_size == 6144. To route through this engine:

1. The model is registered under family "qwen3" in model_list.json.
2. The Qwen3 engine requires model.layers.{i}.self_attn.q_norm.weight and k_norm.weight ($[128]$ BF16). Because MiniCPM5-2B has no QK-normalization, we injected synthetic unit tensors ($\gamma = 1.0$) into model.q4nx across all 42 layers. In RMSNorm: $$\text{RMSNorm}(x, \gamma = 1.0) = \frac{x}{\sqrt{\frac{1}{d}\sum x_i^2 + \epsilon}} \cdot 1.0$$ This satisfies the weight loader but does not preserve the original computation: unit-scale RMSNorm still normalizes Q and K, so attention scores differ from the un-normalized MiniCPM5 architecture. Output equivalence of this port is unvalidated.

---

Installation & Serving Guide

Linux (Ubuntu 24.04 / Linux 6.11+)

1. Clone the Model Repository

Clone directly into your user FastFlowLM models directory:

mkdir -p ~/.config/flm/models
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2 ~/.config/flm/models/MiniCPM5-2B-NPU2

2. Copy Kernel XCLBINs

FastFlowLM requires compiled AIE kernels to reside in its xclbins/ directory. Copy or symlink them:

# Locate your FLM installation root (e.g., ~/.config/flm or /var/lib/lemonade/.cache/lemonade/bin/flm/npu)
FLM_ROOT="${HOME}/.config/flm"
mkdir -p "${FLM_ROOT}/xclbins/MiniCPM5-2B-NPU2"
cp ~/.config/flm/models/MiniCPM5-2B-NPU2/*.xclbin "${FLM_ROOT}/xclbins/MiniCPM5-2B-NPU2/"

3. Register Model in FastFlowLM

Add the model definition to your FastFlowLM model_list.json (located in ~/.config/flm/model_list.json or /opt/flm/model_list.json):

{
  "models": {
    "minicpm5": {
      "2b": {
        "name": "MiniCPM5-2B-NPU2",
        "url": "https://huggingface.co/julianmb/MiniCPM5-2B-NPU2",
        "size": 2023518368,
        "flm_min_version": "0.9.22",
        "files": [
          "config.json",
          "model.q4nx",
          "tokenizer.json",
          "tokenizer_config.json"
        ],
        "default_context_length": 32768,
        "max_prefill_len": 4096,
        "details": {
          "family": "qwen3",
          "think": false,
          "parameter_size": "2B",
          "quantization_level": "Q4_1"
        },
        "label": ["general"],
        "footprint": 1.9
      }
    }
  }
}

4. Start the Server

flm serve minicpm5:2b --host 127.0.0.1 --port 8001

5. Test Inference via cURL

curl -s http://127.0.0.1:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "minicpm5:2b",
    "messages": [
      {"role": "user", "content": "Explain quantum superposition in simple terms with an analogy."}
    ],
    "max_tokens": 150,
    "temperature": 0.2
  }'

---

Windows 11 (AMD Ryzen AI / Strix Halo & Strix Point)

1. Prerequisites

1. Ensure the AMD NPU Driver is installed (part of AMD Ryzen AI Software / AMD Adrenalin).
   • Verify in Task Manager $\to$ Performance: You should see an NPU graph (device named AMD IPU Device or AMD NPU Device).
2. Install FastFlowLM for Windows using the official MSI (flm-setup.msi).

2. Download Model Weights

Open PowerShell and clone or copy the repository into your user profile directory:

New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\flm\models"
cd "$env:USERPROFILE\.config\flm\models"
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2

3. Copy Kernel XCLBINs

Copy the precompiled .xclbin files to your FastFlowLM xclbins directory:

$FLM_DIR = "$env:USERPROFILE\.config\flm"
New-Item -ItemType Directory -Force -Path "$FLM_DIR\xclbins\MiniCPM5-2B-NPU2"
Copy-Item "$FLM_DIR\models\MiniCPM5-2B-NPU2\*.xclbin" -Destination "$FLM_DIR\xclbins\MiniCPM5-2B-NPU2\"

4. Register the Model

Open C:\Program Files\FastFlowLM\model_list.json (or %USERPROFILE%\.config\flm\model_list.json) in a text editor (e.g., VS Code or Notepad with administrator privileges) and add the minicpm5 block under "models":

"minicpm5": {
  "2b": {
    "name": "MiniCPM5-2B-NPU2",
    "url": "https://huggingface.co/julianmb/MiniCPM5-2B-NPU2",
    "size": 2023518368,
    "flm_min_version": "0.9.22",
    "files": [
      "config.json",
      "model.q4nx",
      "tokenizer.json",
      "tokenizer_config.json"
    ],
    "default_context_length": 32768,
    "max_prefill_len": 4096,
    "details": {
      "family": "qwen3",
      "think": false,
      "parameter_size": "2B",
      "quantization_level": "Q4_1"
    },
    "label": ["general"],
    "footprint": 1.9
  }
}

5. Launch the Server via PowerShell

flm serve minicpm5:2b --host 127.0.0.1 --port 8001

6. Test Inference via PowerShell

$body = @{
    model = "minicpm5:2b"
    messages = @(
        @{ role = "user"; content = "Write a Python function to compute Fibonacci with memoization." }
    )
    max_tokens = 200
    temperature = 0.1
} | ConvertTo-Json

Invoke-RestMethod -Uri "http://127.0.0.1:8001/v1/chat/completions" `
    -Method Post `
    -Headers @{ "Content-Type" = "application/json" } `
    -Body $body | Select-Object -ExpandProperty choices | Select-Object -ExpandProperty message

---

Python OpenAI Client Example (Cross-Platform)

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8001/v1",
    api_key="none"  # FastFlowLM local server does not require an auth key
)

response = client.chat.completions.create(
    model="minicpm5:2b",
    messages=[
        {"role": "system", "content": "You are a helpful and concise assistant."},
        {"role": "user", "content": "What are the three laws of robotics?"}
    ],
    max_tokens=200,
    temperature=0.3
)

print(response.choices[0].message.content)

---

Repository Contents

MiniCPM5-2B-NPU2/
├── .gitattributes          # Git-LFS rules for *.q4nx, *.xclbin, tokenizer.json
├── README.md               # Model card & documentation
├── model.q4nx              # 1.88 GB AMD Q4NX weight tensor file
├── config.json             # FastFlowLM NPU SRAM addressing & runtime configuration
├── attn.xclbin             # Precompiled XDNA 2 Multi-Head Attention kernel (d128_q2)
├── mm.xclbin               # Precompiled XDNA 2 Matrix Multiply / SwiGLU kernel
├── layer.xclbin            # Precompiled XDNA 2 Transformer pipeline kernel
├── dequant.xclbin          # Precompiled XDNA 2 Q4_1 dequantization kernel
├── tokenizer.json          # Fast tokenizer vocabulary
├── tokenizer_config.json   # Tokenizer parameters
├── special_tokens_map.json # Special token IDs (BOS, EOS, PAD)
└── chat_template.jinja     # Jinja2 chat template formatting

---

Troubleshooting & FAQ

Q1: [ERROR] Failed to load model: [json.exception.type_error.302] type must be number, but is null

• Cause: FastFlowLM's C++ loader requires explicit integer token IDs in tokenizer_config.json.
• Fix: Fixed in the latest repository commit. Run git pull in your MiniCPM5-2B-NPU2 directory, or verify that your tokenizer_config.json contains:

  "bos_token_id": 0,
  "eos_token_id": [1, 130073],
  "pad_token_id": 1
  

  Note: eos_token_id must be a JSON array [1, 130073] (not a single scalar integer), where 130073 is <|im_end|>.

---

Q2: No such file '.../xclbins/MiniCPM5-2B-NPU2/layer.xclbin'

• Cause: FastFlowLM expects compiled .xclbin kernels inside <flm-root>/xclbins/MiniCPM5-2B-NPU2/.
• Fix: Copy all *.xclbin files from the cloned model folder into your FastFlowLM xclbins/MiniCPM5-2B-NPU2/ directory as described in Step 2 of the guide.

---

Q3: runlist failed execution (ERT_CMD_STATE_TIMEOUT) during Decode

• Symptoms: Prompt prefill succeeds (Prefill chunk 1/1 with ... tokens), but the server immediately returns {"error":"runlist failed execution (ERT_CMD_STATE_TIMEOUT)"} upon entering the autoregressive token generation phase (Start generating...).
• Technical Root Cause: In FastFlowLM's causal LM decode implementation (libqwen3_npu.so), generation dispatches through xrt::runlist (execute() → wait(), per disassembly). One hypothesis under investigation is that all transformer layer forward passes are chained into a single synchronous submission:

  call 24510 <xrt::runlist::execute()@plt>
  call 24710 <xrt::runlist::wait(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) const@plt>
  

  MiniCPM5-2B features 42 transformer layers, whereas the borrowed layer.xclbin and execution graph were compiled for 28 layers (Qwen3-1.7B-NPU2). It is hypothesized — but not yet demonstrated — that submitting 42 sequential layer executions in one chained xrt::runlist may exceed a command-queue depth or timeout threshold. A separate architectural-compatibility question is also open (see §3): the port re-labels a Llama-family model as Qwen3 and injects unit QK-norm scales that still normalize Q/K, so numerical equivalence is unvalidated independent of the dispatch failure.
• Correction regarding -noert: Earlier notes suggested building XRT with -noert. As confirmed in community testing, [-noert] in XRT's build.sh is strictly a build-time flag for building XRT without bundling Alveo PCIe ERT firmware (and is already implied by [-npu]); it is not a runtime scheduler switch and does not bypass the hardware command queue timeout.
• Current Recommendation & Status: Until FastFlowLM upstream implements multi-runlist layer chunking or provides a dedicated 42-layer AIE binary pipeline, 24–28 layer models (such as Qwen3-1.7B-NPU2, Qwen3.5-0.8B-NPU2, and Llama-3.2-1B-NPU2) represent the verified operational ceiling on AMD Strix Halo. We are tracking this limitation with the ROCm / FastFlowLM upstream maintainers.

---

Credits & References

• Base Model: OpenBMB MiniCPM5-2B
• AMD XDNA 2 Runtime: ROCm FastFlowLM
• Research & NPU Experiments: npuhalo (GitHub)