kev-0.8b-rknn

Kev-0.8B for Rockchip RK3588, as an RKLLM file rather than an RKNN graph. The language model is Qwen/Qwen3.5-0.8B-Base with the Kev LoRA merged in, quantized w8a8 by RKLLM-Toolkit 1.3.1. The pointer head stays fp32 NumPy and runs on CPU.

Kev does not generate text. One forward pass reads the last-layer hidden state of every token (RKLLM_INFER_GET_LAST_HIDDEN_LAYER). The score of an option is (k(h_opt) · q(h_decide)) / sqrt(256), with q and k the two Linear(1024 → 256) maps in the .npy files.

Files

librkllmrt.so is not in this repo. The file was exported with toolkit 1.3.1 and loads only with librkllmrt 1.3.1 or newer. Take the aarch64 build from airockchip/rknn-llm (rkllm-runtime/Linux/librkllm_api/aarch64/librkllmrt.so) and load that path in the Kev process. Runtime 1.3.0 rejects this file. Leave any other copy of librkllmrt.so (for example the one serving a different RKLLM model) where it is.

Shape

max_context is 320. A row longer than that does not fit this file. The check prompt below is 292 tokens, so most of the budget is the question and its options, not the state.

Pass token ids with an empty chat template. The runtime must not wrap them in a chat prompt. Use max_new_tokens=0 and keep_history=0. Hidden states come back for every token; if the runtime prepends any, shift the <decide> and </opt> indices by num_tokens - len(ids).

Special tokens in the Kev row are <|fim_prefix|>, <|fim_middle|>, <|box_start|>, <|box_end|>, <|fim_suffix|>. Encoding is the Kev encode path: one causal row of state plus the question, because Gated DeltaNet does not use an attention mask. Packing several questions into that one row lets later questions leak into earlier recurrent state. This file was checked with a single question.

Check

Prompt Панда, дай лапу with the 14-command panda rubric:

The cosine is logit similarity on that one item, not a classification accuracy. The CPU number is the pure PyTorch DeltaNet path, without flash-linear-attention kernels.

Rockchip's own Qwen3.5-0.8B w8a8 table is 588 ms time-to-first-token at 128 tokens on RK3588. At 292 tokens that rate is about 1.3 s. This build did not get under 1 s. optimization_level=0 is the speed setting in Rockchip's benchmark notes; level 1 was slower here (about 1770 ms) and is not the file in this repo.

NPU, CPU, and DDR clocks were already at their maximum during that measurement. The three cores move together, and the prefill is not NPU-bound for the whole 1.7 s.

Provenance

model_lora was tried first. Toolkit 1.3.1 printed Found unsupported layer for every lora_A / lora_B tensor (attention, MLP, and DeltaNet in_proj_* / out_proj) and still wrote a base .rkllm plus a small LoRA sidecar. That pair is not Kev. The file here is the merged model, converted with no model_lora argument.

An ONNX export of the PyTorch DeltaNet is not a usable RKNN graph: the unrolled recurrence was about 140k nodes, and rknn.load_onnx on toolkit 2.3.2 was killed around 27 GB of RAM before rknn.build. Gated DeltaNet in this build is the one inside librkllmrt, not a custom RKNN op.

Convert it again

On an x86 machine with Python 3.10, about 16 GB of RAM free, and the RKLLM-Toolkit 1.3.1 wheel from airockchip/rknn-llm. Install that wheel with --no-deps into an environment that already has a current torch and transformers. The toolkit's own requirements pin older torch and transformers; this export was done with torch 2.14 (CPU) and transformers 5.17 without downgrading them. load_huggingface(..., device="cpu") rejects float16 and must be float32.

python3.10 -m venv .venv
.venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
.venv/bin/pip install 'transformers==5.17.0' peft
.venv/bin/pip install --no-deps rkllm_toolkit-1.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
# pure dependencies the wheel imports: colorlog datasets multiprocess pandas
# python-dateutil pytz six pyarrow dill requests urllib3 xxhash attrs jsonlines
.venv/bin/pip install 'kev @ git+https://github.com/jaredpalmer/kev'

# requests.json is your own list of {"request", "target"} calls. It is not in this repo.
.venv/bin/python scripts/make_calib.py requests.json work/calib.json
.venv/bin/python scripts/merge_lora.py work/merged
.venv/bin/python scripts/build_rkllm.py work/merged work/calib.json kev-0.8b-w8a8-opt0-ctx320.rkllm

merge_lora.py pins the base revision and the adapter revision listed above. build_rkllm.py sets layer_type = block_type on Qwen3_5DecoderLayer before import, because transformers 5.17 renamed the attribute and RKLLM 1.3.1 still reads layer_type.

Copy q_*.npy, k_*.npy, and pointer.json from this repo. They come from head.pt in the adapter repo and are not inside the .rkllm.

License

Apache-2.0 for both the Kev adapter and the Qwen3.5 base, so the merged weights are Apache-2.0. Notices are in NOTICE. librkllmrt.so is Rockchip's runtime (their license on rknn-llm, plus MIT notices for the ggml and llama.cpp pieces) and is not part of this Apache-2.0 distribution.