laya-pdpa-onnx-v2

Superseded. Prefer Suarify/laya-pdpa-onnx-v3-32k. This build was measured on a test split containing zero positive rows for pdpa_mykad, pdpa_bank_card and pdpa_credential_otp, so any accuracy quoted here for those three is the rate a model scores by answering "no" to everything. Kept public for reproducibility, not for use.

Accuracy caveat. The overall figures on this card were measured on a test split whose majority-class baseline is 81.7%, and which contained zero positive rows for pdpa_mykad, pdpa_bank_card and pdpa_credential_otp — so no accuracy quoted here says anything about those three classes. See the retraction on Suarify/laya-pdpa-my-v2. The corpus has since been re-split; prefer the v3 line.

ONNX export of Suarify/laya-pdpa-my-v2, a Malaysian PDPA classifier for phone-call transcripts.

fp32 only. The fp16 and q4f16 variants failed during export and are not published. At 1.29 GB this is usable from Node (onnxruntime-node) but is too large for a browser. See Variants below.

Files

Layout follows the onnx-community convention — plain directories, dtype by filename, one config — rather than a manifest with per-file checksums.

Interface

inputs   input_ids      int64  [batch, seq]
         attention_mask int64  [batch, seq]
         marker_pos     int64  [batch, options]
         marker_mask    bool   [batch, options]
         qtype          int64  [batch]      0=choice 1=score 2=noul
outputs  logits         float  [batch, options]
         act_probs      float  [batch]

All batch axes verified symbolic. Parity against PyTorch: max |diff| 1.32e-04 across batches 1, 3, 8 and 16.

Building the input — get this exactly right

[CLS] <type> question: <instructions> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP]

marker_pos holds the index of each [MASK]; the option logits are read at those positions. Off by one and you get plausible nonsense, not an error.

Special token ids are mmBERT's, not BERT's:

{"cls": "<bos>", "clsId": 2, "sep": "<eos>", "sepId": 1,
 "mask": "<mask>", "maskId": 4, "pad": "<pad>", "padId": 0}

Option text is rendered in label-index order:

• noul → ["false: <crit or 'no, the statement does not hold'>", "true: <crit or 'yes, the statement holds'>"]
• score → ["level 0: <crit[0]>", "level 1: ...", ...]
• choice → ["<key>: <value>"] per criterion, or just <key> when the value is empty

Budgeting: head_max_len 256 covers instructions plus options; each option is capped at 48 tokens; the state is appended last and truncated to whatever room remains under max_len 1024. Note it truncates from the right, so a call longer than 1024 tokens loses its most recent turns.

maxQuestionsPerBatch is 1 — one forward pass per question. There is no sharing between questions, so ten questions cost ten encodes of the same transcript.

Accuracy and speed

accuracy (fp32, 360 decisions)   91.9%    identical to the PyTorch checkpoint
latency  1 question, 4 vCPU      ~1.7 s
latency 10 questions, 4 vCPU     ~15 s
latency 10 questions, GPU        ~150 ms  (calculated, not measured)

Cost is roughly linear in question count and super-linear in transcript length — attention is quadratic, so halving the window is closer to a 3x saving than 2x. This is compute-bound at near CPU peak, not a tuning problem.

Calibration

temperature (choice, score, noul) = [10.0, 10.0, 6.748]

noul is a genuine fit. choice and score sit exactly on the 10.0 clamp, which means the optimiser wanted to soften further and could not — their confidence is unreliable, though their argmax (and so their accuracy) is unaffected.

Variants

Only fp32 was produced. fp16 and q4f16 were attempted and both failed; each variant is built independently so the failures did not cost the fp32 artifact.

For context, per-tensor dynamic INT8 on the v0 checkpoint cost 11 accuracy points (93.0% → 81.9%) for a 4x size win, which is why it is not used here. Block-wise quantisation is the approach both comparable web builds take, and remains the plan.

A browser build is therefore not yet viable. Beyond fixing the variants, 61% of this model is a 256k-row embedding table, and the five supported languages use only 3,968 distinct tokens (1.6%) — pruning the vocabulary to 32k would bring fp16 to roughly 300 MB. That is the path to a browser demo.

Limitations

All training data is synthetic; no real call has ever been labelled. Corpus disclosure prevalence is far above production, so precision is optimistic by roughly an order of magnitude. pdpa_actor scores 50% and should not be gated on. See the base model card for the full set.