laya decision encoders for LiteRT (Android GPU and NPU)

A state in, calibrated answers to typed questions out, no text generated: choice (one of the criteria), score (an ordered scale), noul (does the statement hold, as a probability). One forward per question. These are the convaiinnovations/laya checkpoints (revision 1c5edc17) converted to classic LiteRT graphs with one static window per file: the English model is ModernBERT-large with laya's decision head (421M parameters), the multilingual model is mmBERT-base with the same head (322M; the publisher lists 100+ languages). The request and answer forms are laya's own.

Tested on a Samsung Galaxy S26 (SM-S942Q, Android 16 BP4A.251205.006) with LiteRT 2.2.0 CompiledModel, GPU with explicit FP32 precision, on 2026-09-21. On every captured row the phone's marker scores match the publisher's fp32 Agent.predict (argmax identical, probabilities within 0.0001 after the same calibration), and on the 144 rows of a public three-option decision fixture the phone's choices agree with the publisher's implementation 144/144. Other phones and GPU families have not been tested here.

Which of the three Laya packages to take. All three were tested on a Galaxy S26 GPU with LiteRT 2.2.0 at explicit FP32; the times below are graph time per question at a 256-token window, and the first two packages add the app's table lookup (about 15–20 ms):

• Laya-Multilingual-LiteRT: the multilingual checkpoint (mmBERT-base; the publisher lists 100+ languages; validated in English and Japanese). 0.68 GB on the phone, 51 ms per question. Has an Android sample app with a Kotlin tokenizer (android/; also zero_shot_classification in litert-samples). Take this one for non-English or mixed text; it answers English too, close to the English checkpoint (Convai's XNLI English: 0.843 against 0.860).
• Laya-English-LiteRT: the English checkpoint (ModernBERT-large) and its typed-decisions/ fine-tune for four workflows (customer service, invoice processing, security incidents, agent-trace observability). 0.85 GB each, 123 ms per question (126 ms for the fine-tune). No app of its own; the Kotlin GPU function on its card takes token ids from the app's own tokenizer. Take this one for English-only text when the higher English score matters, and its fine-tune for those four workflows.
• laya-LiteRT: the same two checkpoints (not the fine-tune) in the token-id form: token ids in, embedding table inside the graph (the two packages above take embedding rows that the app looks up from a table file). Its GPU files keep fp32 weights: 1.3 GB (multilingual) or 1.7 GB (English), 54 ms (multilingual) or 127 ms (English) per question with no lookup to add. No app of its own; it carries a reference host (laya_host.py) and HOST_CONTRACT.md, which specifies the sequence and the decoding for a port. Take this one to feed token ids and skip the table lookup in the app.

This repository is the token-id package.

Files

Host files: en/ and multilingual/ hold each checkpoint's exact tokenizer.json, tokenizer_config.json and rl_agent_config.json from the pinned revision; multilingual/calibration.json is a temperature fit made for this conversion (below); laya_host.py is the reference host (laya 0.3.4's prompt builder and decoder, a CPU CompiledModel runner); HOST_CONTRACT.md specifies the sequence and the decoding for a port; SHA256SUMS lists every file.

wfp16 files store the FULLY_CONNECTED and EMBEDDING_LOOKUP weights as float16 with a DEQUANTIZE to float32; activations stay float32 and inputs are int32 ids. On the tested phone LiteRT 2.2.0's GPU accelerator leaves those DEQUANTIZE nodes (and the EMBEDDING_LOOKUP) to the CPU partition and then fails to compile the model, so the wfp16 files were run on the CPU; the fp32 files compile on the GPU. Weight storage and GPU computation precision are separate settings: the tests used explicit FP32 (CompiledModel.GpuOptions(precision = FP32)); the default precision, INT8 and NPU execution were not tested.

fp16-safe revision (2026-09-29) and the NPU. All six graphs were re-exported with two rewrites that give identical results in fp32 (the graphs before and after agree bit for bit in PyTorch and give the same answers on every parity row): each large LayerNorm computes on its input scaled by a power of two with epsilon scaled by the square, because the <bos>, separator and some punctuation rows reach 1.4e4 (multilingual) and 3.3e4 (English) and their squares overflow fp16; and the attention masks use −1e4 instead of −1e9, which is −inf in fp16 and turns a zero mask weight into NaN. The multilingual graphs also move a power of two between the GeGLU gate and the output projection in layers 11–12. The Hexagon NPU computes in fp16; with these rewrites the three S256 files tried there compiled whole on it (JIT, one DispatchDelegate node, 35–59 s on the first launch, cached afterwards), including the in-graph table of the wfp16 file that the GPU rejects. The S512 files were not run on the NPU. An NPU run needs the app's own CompiledModel NPU path (environment with DispatchLibraryDir and CompilerPluginLibraryDir, Accelerator.NPU, the Qualcomm runtime libraries in the APK).

Related packages. The two packages above (Laya-Multilingual-LiteRT and Laya-English-LiteRT), whose apps look up the embedding rows, keep the token table outside the graph (inputs_embeds input, a float16 table file), which lets their float16-weight graphs compile on the same phone's GPU. The graphs here keep the table inside and take token ids.

Graph contract

Main graph, signature serving_default: input_ids [1,N] int32, attention_mask [1,N] float32 (1 real, 0 pad), qtype_onehot [1,3] float32 (choice / score / noul) -> token_logits [1,N] float32, pooled_cls [1,D] float32 (D = 1024 English, 768 multilingual). Act head: pooled_cls [1,D] + feats [1,4] -> act_logits [1,2]. Right padding. The host builds [CLS] <type> question: <instructions> [SEP] [MASK] opt0 [MASK] opt1 … [SEP] <state> [SEP], gathers token_logits at the marker positions, divides by the temperature for the question type and option count, applies softmax, and rounds like the publisher's code. The token embedding table is inside the graph; the tokenizers are external files. laya_host.py reproduces all 611 captured rows (English 209, multilingual 201 per window).

Python

laya_host.py runs the multilingual graph on the CPU with ai-edge-litert and the pinned tokenizer; neither torch nor the laya package is needed.

# pip install transformers==5.17.0 numpy==2.5.3 huggingface_hub==1.32.0 ai-edge-litert==2.1.6
from huggingface_hub import snapshot_download
from laya_host import LayaHost
root = snapshot_download("litert-community/laya-LiteRT", allow_patterns=["laya_ml_s256_wfp16.tflite", "laya_ml_act_head_fp32.tflite", "multilingual/*", "laya_host.py"])
with LayaHost(tokenizer_dir=f"{root}/multilingual", main_graph_path=f"{root}/laya_ml_s256_wfp16.tflite", act_graph_path=f"{root}/laya_ml_act_head_fp32.tflite",
              window=256, head_max_len=256, temperatures_json=f"{root}/multilingual/calibration.json") as host:
    print(host.predict({"message": "同じ注文の代金が二回引き落とされています。差額の返金をお願いします。"},
                       {"department": {"type": "choice", "instructions": "Which department should handle this request?",
                                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, system errors", "sales": "pricing, new contracts", "other": "everything else"}}}))

Conversion and verification

The 2026-09-29 revision adds the fp16-safe rewrites above; its Mac CPU parity equals the previous revision's (multilingual 201/201, English 140/140 and 209/209, all argmax, max Δp ≤ 0.0015 fp16-weight and ≤ 7e-5 fp32). Converted with litert-torch 0.9.3 from laya 0.3.4 (torch 2.12.1, transformers 5.17.0): the encoder plus the two decision-head layers as one graph per window, RoPE tables baked per window, exact-erf GELU kept, the marker gathering and the act features moved to the host. Parity against the publisher's fp32 Agent.predict on a Mac CPU (ai-edge-litert 2.1.6): English 209 rows at window 512 (140 at 256), multilingual 201 rows at each window; every fp32 file within 3e-5 in probability, every wfp16 file within 1.5e-3, argmax identical. The phone numbers are one run per row, the model loaded and warm, one question per call; the load (tokenizer, GPU compile) took 5 to 12 seconds. Later runs in the same session were slower (thermal status 1 at 40 C by the end), so the medians come from the first 60 rows of each run.

The multilingual checkpoint ships no temperature calibration (temperature: [1, 1, 1]). multilingual/calibration.json was fitted for this conversion on 4,415 labeled examples from public datasets (Civil Comments, MASSIVE en-US / ja-JP, JGLUE JNLI / JCommonsenseQA / JSTS, JMTEB livedoor; names, splits and terms are recorded inside the file), minimizing the negative log-likelihood over one temperature per question type and option-count bucket; it leaves every argmax unchanged. The choice:6-10 bucket keeps T = 1 (its fit did not meet the fixed calibration-error allowance). The English checkpoint keeps its upstream temperatures. Calibration on the reader's own data will fit better than this generic one.

Limits

Conversion agreement is not task accuracy: on the 144-row public fixture (three-option evidence questions the model was not trained for) the model's own accuracy was 0.59 multilingual and 0.61 English. The act head's probability saturated at 1.0 on the captured rows and is not a usable escalation signal. A state longer than the window is cut at its end. Japanese states go to the multilingual model with English question schemas, the way the publisher uses it; the English model is for English states.

License and attribution

Apache-2.0 for laya and the English ModernBERT-large encoder (Answer.AI and LightOn); MIT for the multilingual mmBERT-base encoder (JHU CLSP). The converted files are modified derivatives of the laya checkpoints: explicit graph rewrites, static windows, separate act heads, and optional float16 weight storage. LICENSE is laya's Apache-2.0 text; the tokenizer files are unchanged copies. Conversion, calibration fit and Android verification by mlboydaisuke.