feng-30m-zh-tw

A native Traditional-Chinese (zh-TW) variant of feng-30m, a 29.4M-parameter Qwen3-style Chinese chat LLM. The parent is a zh-CN (Simplified) model; this variant reads and writes Traditional Chinese natively.

No fine-tuning, no adapter, no runtime converter. The whole conversion is a deterministic weight-space program that edits one tensor — model.embed_tokens.weight (the tied embedding/unembedding). Every other tensor is bit-identical to the parent, file size, shapes and dtypes are unchanged, the vocabulary stays at 16384, and the tied embedding the parent's on-device C/ESP32 engine requires is preserved.

Demo — same prompt, same greedy decoding, left to right

The animation is real model output (temperature 0, no editing). Red glyphs are Simplified-only, teal glyphs are Traditional-only; the counters are computed live.

Over a fixed 16-prompt × 64-token greedy audit (the prompt list and the counting script are in transform/audit_demo.py; measured values in transform/audit.json, so the numbers below are reproducible against these exact weights):

(The model is 29M parameters: outputs are repetitive and often incoherent. The demo is about script fidelity, not fluency — the parent's quality limits are inherited unchanged.)

How to use it

The tokenizer ships without a chat template, so feed plain strings (the base model continues them; its training format ends the user turn and starts assistant):

from transformers import AutoModelForCausalLM, AutoTokenizer

m = "Luigi/feng-30m-zh-tw"
tok = AutoTokenizer.from_pretrained(m)
net = AutoModelForCausalLM.from_pretrained(m, trust_remote_code=True, dtype="float16").eval()

prompt = "請介紹故宮的歷史。"
ids = tok(prompt, return_tensors="pt")["input_ids"]
out = net.generate(ids, max_new_tokens=64, do_sample=False)[0]
print(tok.decode(out[ids.shape[1]:], skip_special_tokens=True))
# 史前宮是史前宮的官方宮殿，由宮殿、宮殿和宮殿等史前宮殿所共同管理。…

chat_template.jinja is the parent's chat template, kept for tooling compatibility; wrap it yourself if you prefer chat-style prompting. Greedy decoding is recommended — the model is tiny and sampling amplifies its repetition.

Perplexity on seven held-out zh-TW domains

Each slice is 1024 tokens of Traditional Chinese that was never used to choose a knob (domains 3–7 are fresh Wikipedia articles, verified 12-gram-disjoint from the eval slices and from the conversion's attestation corpus).

How zh-TW-ness is measured

Three independent instruments, because no single one is trustworthy alone.

1. Fluency in the target variety — mean cross-entropy on held-out zh-TW text. Primary metric = mean CE (nats/token) over six held-out zh-TW domains, 1024 tokens each, plus per-domain perplexity above. A seventh domain (added late, 38k chars) is a generalization guard: it is measured every run but never optimized. Every candidate edit is scored on all of them, and a knob win that hurts the guard is rejected as slice-overfitting — this is how two knobs that looked optimal on two slices (the word gate 0.78→0.77, the character transfer 0.35→0.65) were re-selected.

2. Purity — the zh-TW rate of what it actually emits. Glyphs are classified with Unihan (s_exclusive = Simplified-only, t_exclusive = Traditional-only), minus the dual-use glyphs the zh-TW attestation corpus proves are legitimately used in Traditional text (台/里/著/学 …).

• Behavioural: generate from a fixed prompt set and count Simplified-only glyphs in the output — this model emits 0 (parent: 19.75 per 100 Han chars, 11/16 prompts leaking).
• Structural: an id is a leak if the string it emits contains a Simplified-only glyph and its row survives at full strength (‖row‖/‖parent row‖ > 0.9). Enforced bound ≤ 200; measured 1 (the corpus-attested dual-use glyph), and 0 unattested Simplified-only emitters.

3. Controls — proving the gain is language identity, not a temperature trick. Two probe corpus files are measured with every structural change:

• zh-CN probe (21k Simplified chars): must not improve. This model: 220 → 769 (deliberately much worse: it is not a zh-CN model any more).
• English probe (22k chars): must not improve materially. This model: 43.60 → 42.81.

The controls caught real traps. Scaling the Simplified-suppression from 0.32 to 0.56 improves mean CE by 0.15% — and drops the zh-CN control from 754 to 394, i.e. it buys perplexity by restoring Simplified emission. That lever (and a global tied-embedding scale, and boosting non-Han rows, which improved English by 4.9%) is a softmax-temperature disguise: it helps any Chinese text, so it is rejected even though every fluency number improves. Conversely the Traditional boost is admissible because its optimum is interior and raising it makes zh-CN worse.

Answer relevance (TMMLU+): what a weight-space edit cannot buy

This model was next asked to answer zh-TW exam multiple-choice questions (TMMLU+). The result is a negative one, and it is the most informative measurement here — so it is documented rather than quietly omitted.

The benchmark's own protocol had to be fixed first. Scoring options one fixed ordering gives 20.5% for this model and 19.5% for the parent, against a 25% chance line. But the prompt lists all four options in context, and the model picks slot 1 41.6% of the time while the answer key is uniform (24.4/29.2/25.6/20.8). A single ordering therefore mixes ranking with a positional artefact, with ±2-question noise on 200. Re-scoring the same questions under all 24 orderings and averaging the option score removes the positional component and most of the noise:

Per-ordering accuracy is at chance for both models: the 29M parent was never trained on exam material, so the questions test knowledge it does not have. Both the reported "+1pp gain" of an earlier configuration and the apparent parent deficit were ordering noise; on the debiased metric the two models are within a point, and every reachable value sits at chance.

Why no edit fixes it. The fixable defects were isolated and attacked with edits far inside any "minority of the weights" budget — none of them adds knowledge:

The conversion's row changes are sharply low-rank (rank-1 = 80.4% of the delta's energy, rank-4 = 85.6%), so a hand-written adapter can carry the conversion in 0.92% of the matrix. It does preserve exam ranking better than overwriting rows (tune 23.6 vs 21.2) — but it cannot represent the row-selective suppression, which is spiky and high-rank, so Simplified emitters return and CE degrades to 5.63 nats, worse than leaving the parent alone (5.51). A hybrid that applies the suppression exactly and factors only the rest recovers tune to 24.4 but still costs CE. Conclusion: this transform is sparse-row surgery by construction, so a low-rank adapter is the wrong basis for it, and low rank does not buy knowledge either.

A variant for users who care about ranking more than perplexity. Setting the character transfer to zero (CHAR_TWIN = 0.0, one knob, no other change) removes the blends that damage option ranking: debiased TMMLU 23.4 → 24.0 (above the parent's 23.2), at a cost of +2.4% mean CE (5.298 → 5.427 nats). The shipped weights use the perplexity-optimal setting; rebuild with CHAR_TWIN = 0.0 for the ranking-optimal one.

A second artifact ships the ranking-optimal side of that trade. Branch tmmlu-optimal contains the same conversion with CHAR_TWIN = 0.0: TMMLU+ debiased 23.8% (vs 23.4% here), mean CE 5.371 (vs 5.298), Traditional-only glyphs 6.09 per 100 Han (vs 9.01), and visibly more repetitive generation. The 0.4pp difference is inside the metric's noise, so the weights in this repo remain the recommended ones.

The honest ceiling. With no exam knowledge in the weights, chance is the ceiling for any admissible row edit, and chance is where both models already are. Beating it requires fine-tuning — which this deliverable deliberately is not.

Weight-space programming for zh-TW nativization

The conversion is written as a program, not a training run. Its signature:

(E_parent, tokenizer, Unihan, zh-TW corpus, knobs)  →  (E_new, renamed_vocab, manifest)

The model's forward pass is never touched; there are no gradients, no data pipeline, no epochs. The search is over edits, not over weights. The interpreter is the model itself: each edit is compiled by running it, and scored by the instruments above. Invariants act as the type system (only embed_tokens may differ; size/shapes/dtype/vocab fixed; the ≤0.9 purity bound), and the program prints every decision it accepts, so the artifact is auditable:

[tw] word-swaps 671  word-renames 2275  word-leaks 1207  char-twins 535
[tw] pieces unique; 0 unassignable ids turned into dead clones
[tw] merges: live in parent 16124, kept after rename 12685 (78.7%)
[tw] TRAD_BOOST 1.1 applied to 4418 native-only emitters, ren-branch 2914
[tw] Simplified-emitting ids: 2879 shadowed (L-scaled), 0 full-strength (word leaks)

The instruction set

All primitives act on a population of ids selected by predicates (glyph sets, corpus attestation, row geometry cos, row norm), and all are parameterized by a scalar:

Because the embedding is tied, each edit simultaneously changes what the model prefers to emit (unembedding) and how it reads a token (embedding) — that coupling is why naive edits fail: suppressing a row harder (to kill a Simplified emission) also destroys the model's ability to read that token in context, which is what makes the tied case genuinely harder than a vocab-swap on the output head.

The two principles that did the work

1. String/row provenance. A token must be served by the row that was trained on the string it now emits. Auditing every place a row serves a foreign string found one violation in the swap branch (a swapped-off Simplified string was being served by the Traditional token's row); fixing it improved all seven domains at once, and degraded the zh-CN control — the signature of a genuine nativization.
2. Corpus attestation. A rename or a glyph transfer is dropped when a 130k-char eval-disjoint zh-TW corpus evidences the other form. This kills the classic per-word dictionary failure modes (e.g. 皇后 → 皇后, 輪廓 → 輪廓) that pure Unihan mapping introduces, and skipping transfers to glyphs the corpus never uses improved every domain.

What measurement closed off

Roughly 125 scored experiments; the negative results are the substance of the method. Row vectors in this model are not composable and not propagatable: nearest-neighbour transfers, char-composition of word rows, word donors for character rows, and vocabulary extension (new ids for untokenised zh-TW words, with merges appended) all measurably hurt, and the character transfer only works from the exact Simplified twin. The softmax geometry is extremely norm-sensitive (a norm-proportional boost is catastrophic: 5.31 → 6.31 nats), so the only usable boost shape is a single constant per population — glyph-count-graded, frequency-graded, norm-proportional and population-split variants were all measured worse. Tokenizer-side ideas are inert by construction: renaming is a decode-side string edit, so the parent's tokenization is untouched and no merge-rule change can move the metric. Suppression algebra is inert too (only its scale matters), because this 16k vocabulary has almost no Traditional single-char tokens — there is no twin row to align a leak to.

The knobs were each swept on the multi-domain mean, and re-selected every time a domain was added — twice, the optimum found on two slices was slice-specific and moved (word gate 0.78→0.77 →0.78 once the character gate was decoupled, character transfer 0.35→0.65→0.40 as the boost and the rename shrink were introduced).

Cost and reproducibility

The final program runs in ~15 s on CPU and 0 GPU-seconds, is deterministic, and records the parent SHA-256, every knob value and the algebra in manifest.json. transform/build_tw.py regenerates this repo bit-for-bit from the parent checkpoint; transform/eval_ppl.py reproduces any number on this page.

pip install safetensors numpy tokenizers transformers
python transform/build_tw.py --base /path/to/feng-30m --out ./feng-30m-zh-tw
python transform/eval_ppl.py --model ./feng-30m-zh-tw --text your_zh_tw_text.txt

Limitations

• Not a good zh-CN model any more — by design (zh-CN PPL 220 → 769). Use the parent for Simplified.
• The parent's esp32/ export is deliberately not copied: it quantises the parent's embedding and is stale here. Re-export from this repo's model.safetensors.
• Gains are domain-dependent: culture/food (val5) is nearly at the parent's level, because the character transfer that makes spelling native costs a little in-domain fluency there.
• Vocabulary and merges are the parent's: ~21% of merge rules leave the reachable set after renaming, untokenised zh-TW words stay character-split (vocabulary extension was measured to break tokenisation), and the parent's own quality ceiling — heavy repetition, weak coherence — is inherited unchanged. This is a script/variety conversion, not a capability improvement.
• No exam/reasoning ability was added. On TMMLU+ both this model and its parent sit at chance on the order-averaged metric (23.4% vs 23.2%, chance 25). Edits inside a 1%–5% weight budget were measured and none of them buys knowledge; the only lever that moved the number at all was removing the conversion's own row transfers, which trades perplexity for ranking (see the answer-relevance section). Adding knowledge is fine-tuning, i.e. out of scope here.
• The conversion writes 42% of the embedding rows (3.1M of 7.3M scalars). That is intentional — renaming strings is row-by-row surgery — but it means this is not a "few-weights" edit; the minority-weight experiments that were tried instead are documented above and did not work.

Lineage / credits

Derived from jiaheng0815/feng-30m (code) — parent model, architecture, tokenizer, Apache-2.0 license. Glyph mapping from Unihan (kTraditionalVariant / kSimplifiedVariant / kTotalScore).