laya-ov-int8

An OpenVINO IR, int8 weight-compressed export of convaiinnovations/laya, the English Laya checkpoint (ModernBERT-large), for fast CPU inference.

Laya answers typed questions about a piece of state in one forward pass, with no text generation: pick one of several options (choice), rate on a scale (score), or answer yes/no (noul). Every answer comes with a calibrated confidence.

Unofficial derivative. Not affiliated with or endorsed by Convai Innovations. The code that serves it is rupeshs/laya-openvino, a fork of NandhaKishorM/laya.

Quantization results

Scored on 90 preset questions (choice, score, noul) against the original torch checkpoint on a 12th-gen Intel CPU:

That is 3.4x faster than torch on a single question at half the size of fp16. The average probability shift is 0.0087 and the largest is 0.145, so leave some margin if you threshold a confidence.

Files

• openvino_model.xml / openvino_model.bin: OpenVINO IR with int8 weights (NNCF weight compression)
• tokenizer/: the ModernBERT tokenizer
• rl_agent_config.json: model config, including the fitted calibration temperatures

Usage

pip install git+https://github.com/rupeshs/laya-openvino.git

from huggingface_hub import snapshot_download
import laya

# Use local_dir rather than the Hub cache: OVAgent patches tokenizer_config.json in place.
snapshot_download("rupeshs/laya-ov-int8", local_dir="laya-ov-int8")

agent = laya.OVAgent("laya-ov-int8")  # device="CPU" by default

state = {
    "from": "user@acme.com",
    "subject": "Duplicate charge on invoice #4411",
    "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.",
}

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which department should handle this request?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, system errors",
            "sales": "pricing, new contracts",
            "other": "everything else",
        },
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this request?",
        "criteria": ["not urgent", "soon", "critical deadline or blocking issue"],
    },
    "churn_risk": {
        "type": "noul",
        "instructions": "Does the user threaten to cancel or leave?",
    },
}

answers = agent.predict(state, questions)["answers"]
print(answers["department"]["choice"], answers["department"]["confidence"])
print(answers["urgency"]["score"])
print(answers["churn_risk"]["noul"])

All three questions are answered in a single forward pass. OVAgent returns the same payload as the torch laya.Agent.

Demos

• Gradio demo: examples/app.py lets you try presets (sentiment, support triage, phishing, LLM guard, moderation, model routing) or your own questions.
• Flappy Bird: flappy-laya-openvino-cpu plays Flappy Bird with this model on CPU. Watch on YouTube.

Limitations

• English only. Accuracy drops sharply on other languages, especially non-Latin scripts, while the reported confidence stays high. Send it English input only.
• int8 changes about 1 in 90 top answers compared with the torch checkpoint.
• For the model itself (question types, training, benchmarks, fine-tuning), see the upstream project NandhaKishorM/laya.

License

Apache 2.0, the same as the original model. Original work by Convai Innovations.