中文Sign up

CMSManhattan/JiRack-Router-Tokenizer-65K

🤗 Hugging Face sourceapache-2.0-other⚠ No SHA-256 weightsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo CMSManhattan/JiRack-Router-Tokenizer-65K ./model-folder
Needs a seeder →

Enjoy — We extend the JiRack Models Ecosystem! 🚀

JiRack Router Tokenizer Pro

JiRack Pro Tokenizer - 347 active language editions of Wikipedia

High-performance custom Byte-Level BPE tokenizer trained on the full Wikipedia dump across multiple languages.

Developed specifically for intelligent routing models and Mixture-of-Experts systems.

Model Details

  • Developer: Konstantin Grabko (JiRack)
  • License: Apache License 2.0
  • Training Data: Full Wikipedia multilingual dump
  • Vocabulary Size: 65,536 tokens
  • Special Tokens: 128 reserved tokens (including 40+ domain routing tokens)

Key Features

  • Correctly placed <|unk|> token at ID 0
  • Full native support for ChatML format (<|im_start|> / <|im_end|>)
  • Large set of specialized routing tokens (__CODING__, __MATH__, __PYTHON__, __SCIENCE__, etc.)
  • Support JiRack Robotics Technology with tags (<|action_start|> / <|action_end|>)
  • Support JiRack vision , images , audio-visual
  • Strong multilingual performance

Basic system and dialogue tokens

"<|unk|>", "<|endoftext|>", "<|padding|>", "<|im_start|>", "<|im_end|>",

Core roles

"<|im_start|>system", "<|im_start|>user", "<|im_start|>assistant",

Additional useful tokens

"<|im_start|>tool", "<|im_start|>function",

Reasoning block

"<|im_start|>thought",

Tool calls

"<|tool_call|>", "<|tool_response|>",

Multimodality and audio-visual block

"<|image|>", "<|video|>", "<|sound|>", "<|voice|>", "<|listening|>", "<|vision|>",

Emotional state (Mood)

"<|mood_happy|>", "<|mood_sad|>", "<|mood_angry|>", "<|mood_neutral|>",

--- FIM (Fill-in-the-Middle) tokens from StarCoder ---

"<|fim_prefix|>", "<|fim_middle|>", "<|fim_suffix|>",

Robotics (trajectory/command boundaries)

"<|action_start|>", "<|action_end|>",

Highlights

  • Architecture: Byte-Level BPE (Byte-level Byte Pair Encoding) which natively prevents Out-Of-Vocabulary (OOV) tokens.
  • Form Factor: Fully wrapped into Hugging Face PreTrainedTokenizerFast with native ByteLevel decoders for clean cyrillic representation.
  • Chat Standard: Out-of-the-box support for ChatML formatting (<|im_start|> / <|im_end|>).
  • Domain Specialization: Pre-baked atomic routing tokens like __CODING__ and __PYTHON__ etc.

CMS Manhattan Service & Support

  • Jirack patent guards your technology for competitors
  • Redesign Llama , Qwen , Gemma to Ternary model
  • Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347
  • Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Router-Tokenizer-65K")

text = "<|im_start|>user\n__CODING__ __PYTHON__ Write a merge sort function.<|im_end|>"
print(tokenizer.encode(text))

JiRack Pretrain Dataset

https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset

python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt:  27%|█████████████████████▋                                                  | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt:  87%|███████████████████████████████████████████████████████████████████████      | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]

Benchmark for tokens quality .

=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>


=== Tokens (IDs) ===
[5, 326, 5095, 944, 396, 23348, 1021, 7831, 5869, 141, 4, 326, 6, 326, 29, 348, 44, 26876, 396, 52698, 6521, 2031, 460, 7524, 141, 4, 326]

=== Decoding Token by Token ===
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
5 -> '<|im_start|>system'
326 -> '
'
5095 -> 'You'
944 -> ' are'
396 -> ' a'
23348 -> ' precise'
1021 -> ' ro'
7831 -> 'uter'
5869 -> ' model'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'
6 -> '<|im_start|>user'
326 -> '
'
29 -> '__CODING__'
348 -> ' '
44 -> '__PYTHON__'
26876 -> ' Write'
396 -> ' a'
52698 -> ' merge'
6521 -> ' sort'
2031 -> ' function'
460 -> ' in'
7524 -> ' Python'
141 -> '.'
4 -> '<|im_end|>'
326 -> '
'

📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

📧 Copyright 2026 CMS Manhattan . All rights reserved