Laya — int8 ONNX (on-device)
An on-device ONNX export of convaiinnovations/laya (Apache-2.0, by ConvAI Innovations / NandhaKishorM). This repo only converts the format. All credit for the model goes to the original authors.
laya-int8.onnx(470 MB): ModernBERT-large encoder + decision head.- Encoder MatMuls: weight-only int8 (MatMulNBits, block 32, symmetric).
- Token embeddings: per-row int8.
- Activations and the head stay fp32.
- Needs ONNX Runtime ≥ 1.21.
- Inputs:
input_ids,attention_maskint64 [B,T];marker_posint64 [B,K];marker_maskbool [B,K];qtypeint64 [B] (choice 0, score 1, noul 2). - Outputs:
logits[B,K] (apply the temperatures inlaya.json), andact_logits. - Sequence layout is the same as
rl_common.build_sequencein the original repo. - Parity with PyTorch fp32 on 12 probes: top-1 agreement 11/12, max probability difference 0.16. The one miss is a near-tie on a probe with a literal
[MASK]in the text.
Used by the Streaks app as its optional on-device model.