Laya Multilingual — CPU-Optimized ONNX Checkpoints
This repository provides high-speed, CPU-optimized ONNX checkpoints for convaiinnovations/laya-multilingual (322M parameters, mmBERT architecture).
These weights are engineered specifically for real-time prompt classification, semantic decision routing, and System One fast-path inference with 0.0 ms network latency on commodity CPUs.
Why CPU-Optimized Checkpoints?
The community WebGPU export (mizchi/laya-multilingual-onnx) is formatted in FP16. On standard CPUs without native half-precision vector execution units (such as x86_64 or ARM Neoverse N1/Cortex-A76), ONNX Runtime emulates FP16 in software, resulting in ~3,900 ms latency per sequence.
This repository provides two calibrated alternatives:
| Checkpoint File | Precision | RAM / Disk | CPU Latency | Optimization & Target Hardware |
|---|---|---|---|---|
model.onnx (Default) / model-int8.onnx |
QInt8 | 325 MB | 250 – 300 ms | Dynamic per-channel quantization. Accelerated via ARMv8.2-A asimddp instructions (sdot/udot) and x86 AVX-512 VNNI. Sensitive layers (scorer, act_head, type_emb) remain in full precision to preserve 100% classification accuracy. |
model-fp32.onnx |
Float32 | 1.29 GB | 550 – 650 ms | Full precision FP32 weights. 100% bit-exact parity with PyTorch. Optimized for ARM NEON and x86 AVX2. |
| Original WebGPU export | Float16 | 617 MB | ~3,900 ms | Emulated via software on CPUs lacking native FP16 instructions. |
Files in this Repository
model.onnx: The selective INT8 model (325 MB). Default for instant, low-latency CPU routing.model-int8.onnx: Explicit alias of the INT8 model.model-fp32.onnx: The full-precision FP32 model (1.29 GB).tokenizer.json&tokenizer/: Fast tokenizer files for Hugging Face tokenizers.rl_agent_config.json: Calibrated temperature scaling and category thresholds.
Quickstart with OpenProxy
OpenProxy automatically detects these models for native, in-process combo decision routing.
1. Download into the default OpenProxy directory
mkdir -p ~/.openproxy/models/laya
huggingface-cli download soyelmismo/laya-multilingual-onnx \
--local-dir ~/.openproxy/models/laya \
--include "model.onnx" "tokenizer.json" "rl_agent_config.json"
Or via direct HTTP curl:
mkdir -p ~/.openproxy/models/laya
HF_BASE="https://huggingface.co/soyelmismo/laya-multilingual-onnx/resolve/main"
curl -L -o ~/.openproxy/models/laya/model.onnx "$HF_BASE/model.onnx"
curl -L -o ~/.openproxy/models/laya/tokenizer.json "$HF_BASE/tokenizer.json"
curl -L -o ~/.openproxy/models/laya/rl_agent_config.json "$HF_BASE/rl_agent_config.json"
2. Run in Docker
Mount the models folder into the container:
services:
openproxy:
image: ghcr.io/soyelmismo/openproxy:latest
ports:
- "8787:8787"
volumes:
- ./config.toml:/etc/openproxy/config.toml:ro
- ~/.openproxy/models/laya:/var/lib/openproxy/models/laya:ro
OpenProxy scans /var/lib/openproxy/models/laya and initializes native in-process inference without any Python dependencies or daemons.
License & Attribution
- Base Model: convaiinnovations/laya-multilingual
- Architecture: mmBERT (322M parameters)
- License: Apache 2.0