laya-pdpa-my-v3

Malaysian PDPA classifier for phone-call transcripts: eight yes/no classes, one ordered severity, and one choice for who disclosed the data.

Read the gain, not the headline

Every figure below is against a majority-class baseline of 74.1% on the 51 held-out conversations. Overall 86.5% is about twelve points of signal, not eighty-six. Quoted alone, pdpa_bank_card at 86.3% reads as respectable; it is in fact below the 88.2% you get by answering "no" to everything.

pdpa_bank_card and pdpa_credential_otp do not work. Six training conversations carried each class. Do not gate on them. An earlier release reported both at 100%, which was true and meaningless: that test split contained no positive examples of either, so a model answering "no" unconditionally scored perfectly.

Known failure modes, all measured

• Phrasing. The same call, same IC, same card, same OTP: pdpa_mykad answers yes at 95% with English surrounding turns and no at 7% with Malay ones. On a third transcript it missed the same IC at 8% that it caught at 98% elsewhere. Not uniformly broken — unreliable, which is harder to work with.
• Severity reverts to its prior. On 113 real Malay conversations screened to contain no disclosures, it called 51% of them medium-or-worse while every binary class answered no. 34% of training rows are severity=high. Never surface severity without the class that justifies it.
• A regex wins twice. On the same split a page of patterns scores 100% on pdpa_email and 94.1% on pdpa_credential_otp.
• Confidence is clamped. Fitted temperatures are (10.0, 10.0, 7.916) and laya clamps to [0.5, 5] at load, so choice and score confidence is uncalibrated in serving.

Why not just prompt an LLM?

On accuracy, you probably should. A frontier model would likely beat this zero-shot, without two dead classes. Cost is not an argument — inference is cheap — and latency favours the LLM, which answers ten questions in one call where this needs ten forward passes.

One argument survives, and for this job it is decisive: the transcript never leaves your infrastructure. Sending a recording containing a MyKad number, a card number and an OTP to a third-party API is a transfer you would have to justify under the law you are trying to comply with. A tool for finding PDPA exposure should not create it. Alongside that, the answer is deterministic and auditable, from weights you hold.

So: regex for fixed formats, this model for the contextual calls a pattern cannot make, an LLM for what is left — and only if your data policy permits sending it.

Results

Held-out test split: 51 conversations, 510 decisions.

By language: bm 98.3% · en 80.8% · manglish 86.7% · ta 86.7% · zh 85.8%

Overall 86.5% (441/510)

At 51 rows per question the 95% interval spans roughly ±15 points. Read these as intervals, not point estimates.

Calibration

temperature (choice, score, noul) = [1.0, 1.0, 1.0]

The training-time fit calibrates on items drawn from the TRAINING skeletons, where the model is never wrong, so it returns the identity. Refit on the validation split before using any confidence — see laya-pdpa-refit-calibration.

Limitations

1. All training data is synthetic. No real call has ever been labelled. Corpus disclosure prevalence is far above production, so precision here is optimistic by roughly an order of magnitude. No threshold derived from this model should reach production untested.
2. Slow on CPU — the transcript is re-encoded once per question and attention is quadratic in its length. Structural, not a tuning problem.
3. Sensitive to phrasing. Identical facts reworded have moved a class probability by an order of magnitude in live probes.

Provenance

An LLM writes conversation skeletons containing only {{SLOT}} placeholders; local deterministic code substitutes fabricated Malaysian identities. Labels are a by-product of substitution, never a model's judgement. No real customer transcript was used at any stage, and none was sent to any external service.