中文Sign up

CMSManhattan/JiRack-Pro-Tokenizer-128K

🤗 Hugging Face sourceroboticsapache-2.0-other⚠ No SHA-256 weightsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo CMSManhattan/JiRack-Pro-Tokenizer-128K ./model-folder
Needs a seeder →

Enjoy — We extend the JiRack Models Ecosystem! 🚀

JiRack Pro Tokenizer 128K

JiRack Pro Tokenizer - 347 active language editions of Wikipedia

High-performance production-grade Byte-Level BPE tokenizer developed as part of the JiRack Ternary Models ecosystem.

This is the Pro version designed for maximum quality, compression, and precision in complex real-world applications.

Open Robot platform

ROBOS HUMANOID Project

Robot dataset example

{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and . ROBOTICS"},{"role":"user","content":"Scene: a table in front of me, with a red mug on it. Command: pick up the mug."},{"role":"assistant","content":"reach(x=0.4,y=0.0,z=0.3)"}]}
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and .ROBOTICS"},{"role":"user","content":"Scene: hand near the mug, distance 2 cm. Command: grasp."},{"role":"assistant","content":"grasp(force=60)"}]}
{"messages":[{"role":"system","content":"You are the robot's brain. Use tags and . ROBOTICS"},{"role":"user","content":"Scene: mug in hand, stable. Command: lift 20 cm."},{"role":"assistant","content":"lift(z=0.2)"}]}

{
 "messages": [
 {
 "role": "system",
 "content": "You are a robot. Use tags and . ROBOTICS"
 },
 {
 "role": "user",
 "content": "Scene: a table in front of me, with a red mug on it. Command: pick up the mug."
 },
 {
 "role": "assistant",
 "content": "reach(x=0.4,y=0.0,z=0.3)"
 }
]
}

Key Features

  • Algorithm: Byte-Level BPE
  • Vocabulary Size: 128,256 tokens — excellent balance between precision and efficiency
  • Multilingual & Technical Strength: Optimized for English, Russian, code, scientific literature, and technical documentation
  • Domain Specialization: Strong performance on programming languages, engineering, robotics, and scientific texts

Special Tokens Support

  • Full ChatML dialogue format (<|im_start|>, <|im_end|>)
  • FIM (Fill-in-the-Middle) support for code generation
  • Rich set of domain routing tokens (__CODING__, __PYTHON__, __ROBOTICS__, __SCIENCE__, etc.)
  • Extended robotics and control tokens

CMS Manhattan Service & Support

  • Jirack patent guards your technology for competitors
  • Redesign Llama , Qwen , Gemma to Ternary model
  • Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347
  • Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Pro-Tokenizer-128K")

text = "__CODING__ __PYTHON__ Write a merge sort function in Python."

tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("Token IDs:", token_ids)

Vocab size: 128256
pad_token_id: 2
eos_token_id: 1

JiRack Pretrain Dataset

https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]

Benchmark for tokens quality .

=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>

=== Tokens (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 30667, 395, 58912, 6643, 2299, 462, 8646, 141, 4, 326]

=== Decoding Token by Token ===
5 -> '<|im_start|>system'
326 -> '\n'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
... 
72 -> '__CODING__'
87 -> '__PYTHON__'
30667 -> ' Write'
58912 -> ' merge'
6643 -> ' sort'

=== GERMAN =====
=== Text nach ChatML-Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|>


=== Token (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 1818, 420, 17119, 4086, 5977, 1039, 140, 178, 800, 140, 165, 11028, 472, 462, 8646, 141, 4, 326]

=== Dekodierung Token für Token ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
1818 -> ' Sch'
420 -> 're'
17119 -> 'ibe'
4086 -> ' eine'
5977 -> ' Mer'
1039 -> 'ge'
140 -> '-'
178 -> 'S'
800 -> 'ort'
140 -> '-'
165 -> 'F'
11028 -> 'unkt'
472 -> 'ion'
462 -> ' in'
8646 -> ' Python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
=== SPANISH =====

=== Texto después de la plantilla ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Escribe una función de ordenamiento por mezcla (merge sort) en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 4230, 12807, 1569, 38999, 444, 16400, 13573, 1441, 15031, 71301, 450, 113333, 6643, 136, 561, 8646, 141, 4, 326]

=== Decodificación token por token ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
4230 -> ' Es'
12807 -> 'cribe'
1569 -> ' una'
38999 -> ' función'
444 -> ' de'
16400 -> ' orden'
13573 -> 'amiento'
1441 -> ' por'
15031 -> ' mez'
71301 -> 'cla'
450 -> ' ('
113333 -> 'merge'
6643 -> ' sort'
136 -> ')'
561 -> ' en'
8646 -> ' Python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'

=== RUSSAIN =====

=== Текст после ChatML шаблона ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|>


=== Токены (ID) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 24549, 57864, 16351, 56848, 101013, 111998, 3263, 945, 1657, 1081, 822, 75733, 141, 4, 326]

=== Декодирование по токенам ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
24549 -> ' Нап'
57864 -> 'иши'
16351 -> ' функ'
56848 -> 'цию'
101013 -> ' сорт'
111998 -> 'ировки'
3263 -> ' сл'
945 -> 'ия'
1657 -> 'ни'
1081 -> 'ем'
822 -> ' на'
75733 -> ' python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'

=== FRANCE =====

=== Texte après le modèle ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|>


=== Tokens (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 112537, 3439, 2834, 27517, 444, 3276, 33659, 561, 8646, 141, 4, 326]

=== Décodage token par token ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
112537 -> ' Éc'
3439 -> 'ris'
2834 -> ' une'
27517 -> ' fonction'
444 -> ' de'
3276 -> ' tri'
33659 -> ' fusion'
561 -> ' en'
8646 -> ' Python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'

=== CHINA =====

=== ChatML 模板处理后的文本 ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|>


=== Token (ID) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 348, 2879, 8646, 348, 22739, 19808, 72775, 15454, 20847, 29714, 115881, 760, 4, 326]

=== 逐个 Token 解码 ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
348 -> ' '
2879 -> '用'
8646 -> ' Python'
348 -> ' '
22739 -> '写'
19808 -> '一个'
72775 -> '归'
15454 -> '并'
20847 -> '排'
29714 -> '序'
115881 -> '函数'
760 -> '。'
4 -> '<|im_end|>'
326 -> '
'

=== JAPAN =========

=== ChatMLテンプレート適用後のテキスト ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Pythonでマージソートの関数を書いてください。<|im_end|>


=== トークン (ID) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 8646, 1183, 3911, 11941, 7947, 7691, 720, 6055, 108920, 6689, 7351, 3176, 5222, 99686, 760, 4, 326]

=== トークンごとのデコード ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
8646 -> ' Python'
1183 -> 'で'
3911 -> 'マ'
11941 -> 'ージ'
7947 -> 'ソ'
7691 -> 'ート'
720 -> 'の'
6055 -> '関'
108920 -> '数を'
6689 -> '書'
7351 -> 'いて'
3176 -> 'く'
5222 -> 'だ'
99686 -> 'さい'
760 -> '。'
4 -> '<|im_end|>'
326 -> '
'

=== ARABIC =======

=== النص بعد تطبيق قالب ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|>


=== الرموز (IDs) ===
[5, 326, 5208, 965, 395, 24704, 1014, 7861, 6124, 141, 4, 326, 6, 326, 72, 348, 87, 45789, 8459, 770, 24238, 6361, 1142, 5451, 5669, 1062, 450, 113333, 6643, 136, 7954, 27188, 88636, 2181, 1343, 141, 4, 326]

=== فك الترميز رمزا برمز ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5208 -> 'You'
965 -> ' are'
395 -> ' a'
24704 -> ' precise'
1014 -> ' ro'
7861 -> 'uter'
6124 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
72 -> '__CODING__'
348 -> ' '
87 -> '__PYTHON__'
45789 -> ' اك'
8459 -> 'تب'
770 -> ' د'
24238 -> 'الة'
6361 -> ' فر'
1142 -> 'ز'
5451 -> ' بال'
5669 -> 'دم'
1062 -> 'ج'
450 -> ' ('
113333 -> 'merge'
6643 -> ' sort'
136 -> ')'
7954 -> ' بل'
27188 -> 'غة'
88636 -> ' باي'
2181 -> 'ث'
1343 -> 'ون'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'

📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

📧 Copyright 2026 CMS Manhattan . All rights reserved