Parity vocabulary pack — ja for HuggingFaceTB/SmolLM2-135M

2 new tokens. No continued pretraining. Certified drift.

Read this before using it: this pack is a demonstration and a negative result. Only 2 of 96 certified candidates cleared a 0.35-nat drift tolerance on this model, so the token reduction it delivers is 0.5% — not a useful saving. The artefact format, the pipeline and the certificate are real and complete; what the measurement shows is that aggressive byte-level Japanese merges on a 135M English-centric model cannot be certified at a useful scale. See the repository README for the full drift study.

This pack adds 2 tokens to HuggingFaceTB/SmolLM2-135M so that ja text costs fewer tokens to read and write. The base model's weights are unchanged: the pack only appends embedding rows, so a request that does not select this pack is served by the original model, bit for bit.

Token cost in ja is a property of the base tokenizer — an artefact fit to a corpus in which most of the world's writing systems were under-represented. This pack repairs that artefact for one language. It says nothing about the language.

What it buys

What it risks — the certificate

Every token in this pack carries a drift certificate measured on held-out calibration contexts, disjoint from the ones its embedding was fitted on.

With probability ≥ 0.95 over the calibration draw, at least 95% of future inputs from the calibration distribution have KL(original ‖ Parity) ≤ 0.09902 nats and total variation ≤ 0.1728, for every token in this pack.

Those are the worst-case values across the pack; per-token bounds are in manifest.json. Tokens whose bound exceeded the build tolerance were not adopted.

Scope of the guarantee. These are finite-sample, distribution-free bounds with respect to the calibration corpus (opus100 (8000 lines, held out)). They are not worst-case over all possible inputs. An adversarial prompt, or a domain far from the calibration data, is outside the guarantee. English and other non-pack languages are outside the guarantee in the other direction — they are unaffected exactly, by construction, not statistically.

Highest-drift tokens in this pack

Use

from parity import serving
router = serving.load("HuggingFaceTB/SmolLM2-135M", packs=["NagaYu/parity-ja-smollm2-135m"])
print(router.encode("...", view="ja"))   # fewer tokens
print(router.encode("...", view="base"))     # the original tokenizer, unchanged

Build provenance

Contributing a pack for your language

See docs/contributing-a-pack.md in the Parity repository. In short: point the CLI at a corpus you trust for your language, review the mined tokens (they are printed as strings, not ids), and open a pull request with the resulting pack. Review of the token list by speakers of the language is part of the process, not an optional extra.