defuss-lid CLD3 DNO-PQ packs

Google CLD3 strongly compressed and massively improved.

Two CLD3 language-identification packs, major_20 and full, made from Google's CLD3 weights by my new DNO-PQ (Direct Neural Optimization - Product Quantization) method: the embeddings are product-quantized, and the hidden and softmax layers were refitted on held-out text, starting from Google's weights, converted to int8. The packs run on the unmodified CLD3 WebAssembly build of cld3-asm 4.0.0.

Unlike typical compressed models, that suffer from quality loss after quantization and product quantization, this model improves massively over the base model FP32. Compared with Google's original CLD3:

• both packs are 62% to 66% smaller
• and make 14% to 90% fewer labeling errors, depending on the language measured

For two-word starts, which is a significantly harder task for language identification than longer sentence LID, this model makes 49% fewer errors than Google's original with major_20 (80.88% to 90.23% accuracy) and 19% fewer with full (69.38% to 75.26%) on the benchmark.

On two-word starts of other held-out text, the errors fall by 61% and 24% on the Tatoeba test split and by 33% and 14% on NTREX-128 news.

Runtime

The model packs are compatible with the defuss-lid CLD3 runtime:

Github: defuss-lid

The runtime has been tested in-browser, in Node.js and with bun.

Packs

Size compared with Google's CLD3

Measured 2026-11-03; gzip at level 9 with fflate, as in the source repository's size table. "Google original" is Google's released CLD3 weights in the same container, unchanged; "all-in" adds what an application downloads besides the pack: the CLD3 pack runtime JavaScript and the shared WebAssembly core (188,549 bytes, 108,233 gzipped).

Usage

The packs load with createCld3Pack / loadCld3Pack from defuss-lid/cld3-pack and need the shared WASM core cld3-core.bin.gz, which the npm package ships as runtime/cld3-core.bin.gz (the loader verifies its SHA-256):

import {readFile} from 'node:fs/promises';
import {createCld3Pack} from 'defuss-lid/cld3-pack';

const lid = await createCld3Pack(await readFile('full.cld3q'), {core: await readFile('cld3-core.bin.gz')});
console.log(lid.predict('Bonjour tout le monde.'));  // [{language: 'fr', score: 1.92…, reliable: true}]
console.log(lid.predict('Dzień dobry')[0].score);  // 1.99…
console.log(lid.predict('Dzień dobry', {boost: false})[0].score);  // 0.99…, the model alone
lid.dispose();

Letter boost: Letter boosting is a defuss-lid algorithm enhancement that further improves the results slightly: the score is the model's probability plus the share of the text's letters the language's alphabet explains (up to 1), and a language whose alphabet alone explains the letters better than the answer's boosted score replaces it. {boost: false} returns the model's own probability.

In a browser, loadCld3Pack(packURL, {coreURL}) fetches both files.

Measured accuracy

Top-1 accuracy (%), measured with the real WebAssembly build (dnp-tools/cld3/eval.mjs). "Google original" is Google's released CLD3 weights restricted to the same labels, run without letter boost as Google's CLD3 has none. "DNO-PQ" is the pack as defuss-lid runs it by default, with the letter boost: each language's share of the text's letters is added to the model scores. "Errors vs Google" is the relative change in the error rate, (DNO-PQ errors − Google errors) / Google errors; −50% means half the errors. Scores are not calibrated probabilities.

• Benchmark (the defuss-lid benchmark cohorts): FLORES-200 devtest sentences (1,012 per language), their first 20 code points, MASSIVE 1.1 test utterances, and two-word starts: the distinct first two words of every FLORES and MASSIVE benchmark sentence per language (Unicode word boundaries, so Chinese, Japanese and Thai get words too). No benchmark text was used to fit or select the edit.
• Tatoeba test split: 20% of each language's Tatoeba sentences by sentence id, held out from the edit's Tatoeba fit and selection splits.
• NTREX-128: up to 1,000 news sentences per language from a source no fit or selection text came from.
• full only: FLORES-200 devtest of the 19 CLD3 languages outside the benchmark cohort, and synthetic romanized text for the six -Latn labels.

major_20 (20 languages)

full (all 109 CLD3 labels)

Two-word start errors by cause

The remaining wrong two-word starts of the DNO-PQ packs without letter boost, each filed under its first matching cause by scripts/sentence_starts.mjs (which picks its own example sentence for a few duplicate starts, so its totals differ from the tables above by at most 2 starts).

Per-language losses against Google's original

Measured on the answers without letter boost, the only per-language figures recorded.

major_20: no language loses more than 0.9 points on any slice (Japanese on NTREX two-word starts; Italian 0.7 on FLORES first 20 code points).

full: most languages gain, some lose. The losses of more than 1 point:

• Serbian: −9.6 points on NTREX sentences, mostly from the 353 of 1,000 NTREX Serbian sentences that put Latin look-alike letters inside Cyrillic words; on ordinary text −2.8 on Tatoeba sentences, −3.8 on FLORES first 20 code points and −4.7 to −6.9 on two-word starts.
• Croatian: −3.8 on FLORES sentences, −1.8 on Tatoeba, −4.0 on NTREX sentences, −3.8 and −3.7 on FLORES and NTREX first 20 code points, −1.8 to −7.6 on two-word starts (CLD3 confuses it with Bosnian and Serbian; Google's original gets 57.5% of its FLORES sentences).
• Malay: −5.7 on FLORES sentences, −2.4 on MASSIVE, −7.3 on Tatoeba, −4.7 on NTREX, −1.1 to −6.2 on first 20 code points and two-word starts (CLD3 confuses it with Indonesian).
• Norwegian: −2.7 on MASSIVE, −3.3 on FLORES devtest, −6.6 on Tatoeba, −6.1 on NTREX (to Danish, which gains).
• Slovenian: −2.8 on FLORES first 20 code points, −1.2 on NTREX sentences, −7.0 and −6.0 on NTREX first 20 code points and two-word starts.
• Marathi: −3.2 on benchmark and NTREX two-word starts, −2.5 on NTREX first 20 code points.
• Zulu: −1.6 on FLORES devtest, −2.6 on NTREX (CLD3 confuses it with Xhosa).
• Hmong: −4.2 on NTREX first 20 code points.
• 1 to 3 points: Ukrainian (−2.3 on Tatoeba, −1.1 on its first 20 code points, −2.2 on NTREX two-word starts), Dutch (−1.3 and −3.0 on benchmark and NTREX two-word starts), Javanese and Spanish (first 20 code points), Tagalog (NTREX), Bulgarian (NTREX two-word starts), Hawaiian and Azerbaijani (Tatoeba first 20 code points).
• Small Tatoeba test sets: Nyanja (10 sentences), Shona (11), Corsican (13) and Zulu (16) lose one to three sentences or starts each, 6 to 30 points.

How the packs were made

• Changed: all 39,629 hidden- and softmax-layer parameters of full (21,028 of major_20), 18% and 11% of the network's float parameters. The 178,520 embedding values are unchanged. The hidden layer moved far from Google's weights (272% relative L2 change in full, measured against the int8 copy the edit started from), so this is a refit of the two dense layers on frozen embeddings, not a sparse patch.
• Fit data: FLORES-200 dev, MASSIVE 1.1 train and dev, Tatoeba fit and selection splits for every CLD3 label Tatoeba has, and synthetic romanized text. Each sentence also enters as its first 1, 2, 3 and 5 words and its first 10, 20 and 40 code points.
• Constraints: L2 pull towards the starting weights (λ = 1e-6), long texts the starting pack gets right anchored half to its own answer, confusable pairs (hr/bs, da/no, id/ms, xh/zu) keeping the starting pack's split, and a per-language guard with a paired significance test on the selection split; a pack is written only when no language loses there.
• Quantization: both layers stored as int8 with one scale per column, the softmax refitted on the rounded hidden layer's activations.

Limitations

• The losses listed above are real trade-offs; evaluate on your own distribution.
• The romanized test text used to align the direct neural edits was synthetically generated; Google's float32 CLD3 gives it the intended label for 57.6% (zh-Latn) to 98.2% (el-Latn) of sentences.
• Several Tatoeba test sets are small (24 labels under 100 sentences).
• Tatoeba publishes unversioned weekly exports; a rebuild reproduces these packs only from the exports downloaded.
• No unsupported-language rejection and no code-switching accuracy is claimed.
• CLD3 packs rebuild the native WASM layout at load: they reduce download size, not inference RAM.

Licenses and attribution

Apache-2.0, like Google CLD3 (Copyright 2016 Google Inc.); see NOTICE.md. Alignment data, not redistributed: FLORES-200 dev (CC BY-SA 4.0), MASSIVE 1.1 train and dev (CC BY 4.0), Tatoeba (CC BY 2.0 FR). Test data only: NTREX-128 (CC BY-SA 4.0), FLORES-200 devtest, MASSIVE test.