laya-pdpa-onnx-v3-32k

Why not just prompt an LLM?

On accuracy, you probably should. A frontier model would likely beat 83% on this task zero-shot, without two dead classes. Cost is not an argument either -- inference is cheap now -- and latency favours the LLM, which answers all ten questions in one call where this needs ten forward passes.

There is one argument, and for this job it is decisive:

The transcript never leaves your infrastructure. Sending a call recording containing a MyKad number, a card number and an OTP to a third-party API is itself a transfer you would have to justify under the law you are trying to comply with. A tool for finding PDPA exposure should not create it.

Alongside that: the answer is deterministic and auditable -- same input, same output, forever, from a 192 MB file you hold -- where a hosted model can change beneath you between audits. And it runs with no network at all, including in a browser tab.

The sensible design is a cascade, not a replacement.

Measured on the same 51 conversations, a page of regexes scores 100% on pdpa_email against the model's 94.1%. Where the model wins on pdpa_mykad (96.1% vs 76.5%) it wins on precision: the regex fires on order ids, tracking numbers and invoice amounts, which is what the corpus's 16 hard-negative slots were built to teach.

Read the gain, not the headline

Accuracy is quoted against a majority-class baseline of 74.1% on 51 held-out conversations. So 83.1% is about nine points of signal, not eighty-three. Per question, gain over always answering with the most common label:

Two questions do not work

pdpa_bank_card and pdpa_credential_otp perform at or below a constant "no". Each had six training conversations carrying the class. Do not gate on them. An earlier release reported both at 100%, which was true and meaningless: that test split contained no positive examples of either, so answering "no" unconditionally scored perfectly.

Three further limits

• It is sensitive to phrasing. The same call, same IC, same card, same OTP, with the surrounding turns in English versus Malay: pdpa_mykad answers yes at 95% on one and no at 7% on the other.
• Severity reverts to its training prior on unfamiliar text. On 113 real Malay conversations screened to contain no disclosures, it called 51% of them medium-or-worse while every binary class answered "no". Never surface severity without the class that justifies it.
• A regex beats this model on two classes. On the same test split a page of patterns scores 100% on pdpa_email and 94.1% on pdpa_credential_otp. The model's advantage on the digit classes is precision against near-misses — deciding whether a twelve-digit number is a card or an order id — not detection.

All training data is synthetic; no real call has ever been labelled. Validate on your own traffic before trusting any of this.

ONNX export of Suarify/laya-pdpa-my-v3 — a Malaysian PDPA classifier for call transcripts. Ten typed questions per call: eight yes/no classes, one ordered severity, one choice for who disclosed.

Exported with torch.onnx.export(dynamo=True), opset 18, external data.

Variants

Parity against the PyTorch checkpoint: 8.30e-05 max absolute logit difference across batches 1, 3, 8 and 16.

Pruned vocabulary

The base checkpoint carries a 256,000-token multilingual vocabulary, which made the embedding table 61% of its weights. MatMulNBitsQuantizer skips Gather, so quantisation never touched it and the full-vocab q4f16 build still weighed 880 MB.

This export keeps 32,000 tokens, chosen on the train and validation splits only, and slices the embedding to match: 256,000 -> 32,000 rows, 322M -> 150M parameters.

embedding   fp32      98 MB    fp16     49 MB
everything else   fp32     501 MB    4-bit     70 MB

The keep-set is script-aware. Ids below 32,000 hold 28,564 Latin tokens but only 571 CJK and 27 Tamil, so keeping the lowest ids would leave Chinese and Tamil encoding as raw bytes. Every single-character CJK and Tamil token is kept, and byte fallback is retained, so no input is ever unencodable.

The tokenizer in this repo is not interchangeable with the base one. Token ids differ. Use tokenizer.json from here or every answer is read from wrong rows.

Inputs

input_ids       int64  [batch, seq]
attention_mask  int64  [batch, seq]
marker_pos      int64  [batch, options]   index of each [MASK] slot
marker_mask     bool   [batch, options]
qtype           int64  [batch]            0=choice 1=score 2=noul
->  logits      float  [batch, options]

This is not a standard sequence classifier and transformers.js cannot load it. The sequence must be built as laya.common.build_sequence does:

[CLS] <type> question: <ins> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP]

with marker_pos carrying each [MASK] index. Special tokens are mmBERT's <bos>/<eos>/<mask>/<pad> = 2/1/4/0, not [CLS]/[SEP]. Option order is label order — get it wrong and every answer is silently relabelled, not an error. A working single-file browser port is in the web/ directory of the source repo.

Segmentation is unchanged for text the vocabulary was fitted on, and near-identical on held-out text; the export log for this run prints both figures.

Calibration

temperature (choice, score, noul) = [10.0, 10.0, 7.915902137756348]

choice and score sit exactly on the 10.0 clamp: the optimiser wanted to soften further and could not, so those logits are saturated and their confidence is unreliable. Accuracy is unaffected — temperature does not move argmax. noul fitted interior to the range and is genuinely calibrated.

Limitations

All training data is synthetic; no real call has ever been labelled. Disclosure prevalence in the corpus is far above production, so precision here is optimistic by roughly an order of magnitude. pdpa_actor is at 50% and should not be gated on. See the base model card for the full list.