laya-pdpa-onnx-v2-32k

Superseded. Prefer Suarify/laya-pdpa-onnx-v3-32k. This build was measured on a test split containing zero positive rows for pdpa_mykad, pdpa_bank_card and pdpa_credential_otp, so any accuracy quoted here for those three is the rate a model scores by answering "no" to everything. Kept public for reproducibility, not for use.

Accuracy caveat. The overall figures on this card were measured on a test split whose majority-class baseline is 81.7%, and which contained zero positive rows for pdpa_mykad, pdpa_bank_card and pdpa_credential_otp — so no accuracy quoted here says anything about those three classes. See the retraction on Suarify/laya-pdpa-my-v2. The corpus has since been re-split; prefer the v3 line.

ONNX export of Suarify/laya-pdpa-my-v2 — a Malaysian PDPA classifier for call transcripts. Ten typed questions per call: eight yes/no classes, one ordered severity, one choice for who disclosed.

Exported with torch.onnx.export(dynamo=True), opset 18, external data.

Variants

Parity against the PyTorch checkpoint: 1.28e-04 max absolute logit difference across batches 1, 3, 8 and 16.

Pruned vocabulary

The base checkpoint carries a 256,000-token multilingual vocabulary, which made the embedding table 61% of its weights. MatMulNBitsQuantizer skips Gather, so quantisation never touched it and the full-vocab q4f16 build still weighed 880 MB.

This export keeps 32,000 tokens, chosen on the train and validation splits only, and slices the embedding to match: 256,000 -> 32,000 rows, 322M -> 150M parameters.

embedding   fp32      98 MB    fp16     49 MB
everything else   fp32     501 MB    4-bit     70 MB

The keep-set is script-aware. Ids below 32,000 hold 28,564 Latin tokens but only 571 CJK and 27 Tamil, so keeping the lowest ids would leave Chinese and Tamil encoding as raw bytes. Every single-character CJK and Tamil token is kept, and byte fallback is retained, so no input is ever unencodable.

The tokenizer in this repo is not interchangeable with the base one. Token ids differ. Use tokenizer.json from here or every answer is read from wrong rows.

Inputs

input_ids       int64  [batch, seq]
attention_mask  int64  [batch, seq]
marker_pos      int64  [batch, options]   index of each [MASK] slot
marker_mask     bool   [batch, options]
qtype           int64  [batch]            0=choice 1=score 2=noul
->  logits      float  [batch, options]

This is not a standard sequence classifier and transformers.js cannot load it. The sequence must be built as laya.common.build_sequence does:

[CLS] <type> question: <ins> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP]

with marker_pos carrying each [MASK] index. Special tokens are mmBERT's <bos>/<eos>/<mask>/<pad> = 2/1/4/0, not [CLS]/[SEP]. Option order is label order — get it wrong and every answer is silently relabelled, not an error. A working single-file browser port is in the web/ directory of the source repo.

Segmentation is unchanged for text the vocabulary was fitted on, and near-identical on held-out text; the export log for this run prints both figures.

Calibration

temperature (choice, score, noul) = [10.0, 10.0, 6.747796535491943]

choice and score sit exactly on the 10.0 clamp: the optimiser wanted to soften further and could not, so those logits are saturated and their confidence is unreliable. Accuracy is unaffected — temperature does not move argmax. noul fitted interior to the range and is genuinely calibrated.

Limitations

All training data is synthetic; no real call has ever been labelled. Disclosure prevalence in the corpus is far above production, so precision here is optimistic by roughly an order of magnitude. pdpa_actor is at 50% and should not be gated on. See the base model card for the full list.