ja-NCM is one of the first open-weight Japanese Next-Concept Model (NCM) foundation models: a 24x1024 NCM generator and concept tokenizer trained from scratch on Japanese web text, plus translation and summarisation fine-tunes. It applies Next-Concept Modeling (Kumbong et al., 2026, code MIT) to Japanese.
Source Code: https://github.com/ronantakizawa/ja-ncm
Most language models write one token at a time, so a 128-token passage takes 128 forward passes. An NCM works in two parts instead. A concept tokenizer (a multiscale VQ-VAE) compresses a 128-token window into 12 levels of discrete codes: the first level holds the coarse gist of the passage and each later level adds finer detail. A generator transformer then predicts these codes level by level, coarse to fine, and predicts every code within a level at once. A whole passage therefore takes 12 forward passes regardless of its length, and the tokenizer's decoder turns the final codes back into text.
The NCM paper claims that NCM beats diffusion language models, the main family of parallel text generators, on quality at few steps, on wall-clock speed and on training efficiency. Those results are in English. We trained same-budget baselines on the same Japanese data to test the claims in Japanese:
Every model is scored the same way (generative perplexity at matched sample entropy) and timed on the same GPU with the same optimisations.
| Folder | Model | Size | Notes |
|---|---|---|---|
v3/uncond/ |
Unconditional Japanese generator | 24x1024 (316M) | trained from scratch, 100k steps x 512 x 128 tokens |
v3/en-ja/ |
English->Japanese translation | 24x1024 | fine-tuned from v3/uncond |
v3/summ/ |
Japanese summarisation | 12x768 | unconditional pre-training, distilled summaries, then XL-Sum ja |
phase-a/ja-en/ |
Japanese->English translation | 12x768 | released WMT14 De->En NCM fine-tuned on ja->en, frozen English tokenizer |
phase-a/ja-en-ar-baseline/ |
AR baseline for ja->en | 12x768 | decoder-only, same data and schedule |
The three v3 models share v3/tokenizer.safetensors (llm-jp-3 vocabulary, multiscale VQ-VAE,
12 levels, 99.73% token reconstruction). phase-a/ja-en/ has its own tokenizer.
v3 unconditional generation, generative perplexity under llm-jp-3-1.8b at the data's character entropy (lower is better; real text 21.3): NCM (12 steps) 300; same-size masked diffusion (MDLM) 313 at 16 steps and 121 at 128 steps; 12x768 AR 52. Latency at batch 16 on one H100, with CUDA graphs for every model: NCM 6.1 ms/sequence, MDLM-16 14.9 ms, AR 16.9 ms.
v3 en->ja (greedy):
| Test | BLEU (ja-mecab) | COMET-22 | ms/sentence (H100, batch 16) |
|---|---|---|---|
| FLORES+ | 12.16 | 68.36 | 18.3 |
| NTREX | 13.60 | 67.49 | 17.8 |
| KFTT | 18.97 | 73.62 | 18.3 |
| ALT | 19.20 | 72.94 | 18.6 |
v3 summarisation, XL-Sum ja test (MeCab ROUGE-1/2/L): 34.50 / 12.58 / 24.93, 12.9 ms/summary.
Per-test-set scores and outputs are in each folder's results/. Phase A scores are in
phase-a/*/results/.
FineWeb-2-Edu-Japanese sample_10BT (ODC-By); llm-jp-3 tokenizer vocabulary (LLM-jp); JapaneseSummarization-FW2EduJa-Distill (Apache-2.0); XL-Sum ja (CC BY-NC-SA 4.0); KFTT (CC BY-SA 3.0); Mitsua Wikidata (CC0); Tatoeba (CC BY 2.0 FR); ALT (CC BY 4.0); FLORES+ and NTREX-128 for evaluation only. Upstream NCM code and released checkpoints: MIT
Use the ja-NCM project code (vendored NCM model classes). Each folder's config.yaml is the
full training config; weights load with safetensors.torch.load_file and
load_state_dict(strict=True). For a v3 model, point checkpoints.tokenizer at
v3/tokenizer.safetensors.
v3/baselines/uncond-mdlm/model.safetensors 1.4 GBv3/en-ja/generator.safetensors 1.3 GBv3/uncond/generator.safetensors 1.3 GBv3/tokenizer.safetensors 810 MBphase-a/ja-en/tokenizer.safetensors 646 MBphase-a/ja-en-ar-baseline/model.safetensors 576 MBv3/baselines/uncond-ar/model.safetensors 496 MBphase-a/ja-en/generator.safetensors 481 MBv3/summ/generator.safetensors 366 MB