Loading…
Loading…
Added DeekSeek resoning tags
So Do not use tool in system prompt that kills context size see what people do below !
Added Tool calls tags ( "<|tool_call_start|>", "<|tool_call_end|>", "<|tool_result_start|>", "<|tool_result_end|>", )
Size { Vocab size: 151779 pad_token_id: 151643 eos_token_id: 151643 }
Compatible with Qwen 2.5 and DeekSeek R1 • Optimized for Code . Use resize function without adaptation, see examples below
It needs 100k example to fully adapt routing features for RAG Routing model . So check Qwen modification tokenizer rules
A DeekSeek R1-based tokenizer enhanced with FIM markers from Microsoft datasets and other tags for ML ( <|fim_prefix|>, <|fim_middle|>, <f|im_suffix|> )
Added Robotics & Embodiment tags ( "<|action_start|>", "<|action_end|>", "<|trajectory_start|>", "<|trajectory_end|>", "<|joint_start|>", "<|joint_end|>", "<|sensor_start|>", "<|sensor_end|>", "<|command_start|>", "<|command_end|>", "<|state_start|>", "<|state_end|>", "<|pose|>", "<|velocity|>", "<|force|>", "<|torque|>", "<|gripper|>", "<|navigation|>", "<|obstacle|>", "<|task_start|>", "<|task_end|>", "<|plan_start|>", "<|plan_end|>", "<|behavior_start|>", "<|behavior_end|>", "<|skill_start|>", "<|skill_end|>", "<|motor|>", "<|servo|>", "<|imu|>", "<|lidar|>", "<|camera|>", "<|depth|>", "<|waypoint|>", "<|path|>", "<|collision|>", "<|grasp|>", "<|release|>", "<|homing|>", "<|emergency_stop|>", "<|calibration|>", "<|manipulation|>", "<|locomotion|>", "<|feedback|>", "<|control_loop|>",)
Added Multi models support ( "<|image|>", "<|video|>", "<|sound|>", "<|voice|>", "<|listening|>", "<|vision|>",)
Added Human mood tags ( "<|mood_happy|>", "<|mood_sad|>", "<|mood_angry|>", "<|mood_neutral|>", )
Added Tool calls tags ( "<|tool_call_start|>", "<|tool_call_end|>", "<|tool_result_start|>", "<|tool_result_end|>", )
Added RAG routing tags for RAG MoE Systems ( "SCIENCE", "CODING", "STOCK_EXCHANGE", "MEDICINE", "GOVERNMENT", "NEWS", "GENERAL", "MATERIAL_SCIENCE", "ELECTRONICS", "MICROELECTRONICS", "ENGINEERING", "ROBOTICS", "ENERGY", "AUTOMOTIVE", "AVIATION", "MATH", "PYTHON", "C", "CPP", "C_SHARP", "JAVA", "JAVASCRIPT", "TYPESCRIPT", "RUST", "GO", "RUBY", "PHP", "SWIFT", "KOTLIN", "BASH", "SQL", "ASSEMBLY", "PHILOSOPHY", "LITERATURE", "SOCIOLOGY", "PSYCHOLOGY", "POLITICAL_SCIENCE", "CULTURAL_STUDIES", "ETHNOGRAPHY", "HUMAN_RIGHTS", "COMPLIANCE", "MILITARY", "BANKING", "OIL_INDUSTRY", "LIGHT_INDUSTRY", "NATURE", "OCEAN", "SPORT", "CULINARY", "TRAVEL", "HOBBY" )
Fully compatible with Microsoft BigCode datasets including The Stack, StarCoder, and NextCoder.
Enables efficient training on large-scale coding data for superior code generation and understanding.
Inventor: Konstantin Vladimirovich Grabko
Organization: CMS Manhattan JiRack Technology
Official Site: www.cmsmanhattan.com
Designed for Banking and Fintech Institutions
Banks and Fintech JiRack Architecture: Build Sovereign Financial Models from Scratch
JiRack Corp Tokenizer solution
JiRack Tokenizer Subcription
{"messages":[{"role":"system","content":"Ты — мозг робота. Используй теги и . ROBOTICS"},{"role":"user","content":"Сцена: стол передо мной, на нём красная кружка. Команда: возьми кружку."},{"role":"assistant","content":"reach(x=0.4,y=0.0,z=0.3)"}]}
{"messages":[{"role":"system","content":"Ты — мозг робота. Используй теги и . ROBOTICS"},{"role":"user","content":"Сцена: рука у кружки, расстояние 2 см. Команда: сожми."},{"role":"assistant","content":"grasp(force=60)"}]}
{"messages":[{"role":"system","content":"Ты — мозг робота. Используй теги и . ROBOTICS"},{"role":"user","content":"Сцена: кружка в руке, устойчиво. Команда: подними на 20 см."},{"role":"assistant","content":"lift(z=0.2)"}]}
{
"messages":
[
{
"role": "system",
"content": "You are a robot's brain. Use tags. ROBOTICS"
},
{
"role": "user",
"content": "Scene: a table in front of me, with a red mug on it. Command: pick up the mug."
},
{
"role": "assistant",
"content": "reach(x=0.4,y=0.0,z=0.3)"
}
]
}
Here's a concrete example of the Qwen 2.5 / Hermes-style tool-calling format: https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1
System message (defines available tools):
<|im_start|>system
You are a helpful assistant with access to the following functions. Use them if required:
<tools>
{
"type": "function",
"function": {"name": "get_weather", "description": "Get current weather for a location",
"parameters":
{"type": "object", "properties":
{
"location":{"type": "string", "description": "City name"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]
}
}
}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
<|im_end|>
<|im_start|>user What's the weather like in Tokyo right now?<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "get_weather", "arguments": {"location": "Tokyo", "unit": "celsius"}}
</tool_call>
<|im_end|>
<|im_start|>tool
<tool_response>
{"temperature": 22, "condition": "clear sky", "humidity": 48}
</tool_response>
<|im_end|>
<|im_start|>assistant It's currently 22°C and clear in Tokyo, with 48% humidity.<|im_end|>
That's the exact skeleton your Hermes dataset entries need to be mapped into
— system prompt with <tools>...</tools>, <tool_call> in assistant turns, <tool_response> in tool turns.
If you apply Qwen 2.5's tokenizer chat template via apply_chat_template() with a tools=[...] argument,
it generates this structure automatically — you don't have to hand-write the tags yourself for training data prep,
as long as your data is in the standard role/content/tool_calls message-list format first
__CODING__, __PYTHON__, __ROBOTICS__, __SCIENCE__, etc.)Install for Llamma compatible models in your chat script
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(your model)
model.resize_token_embeddings(len(tokenizer.tokenizer)) # или просто len(tokenizer.tokenizer)
print("New Embedding size for you chat script:", model.get_input_embeddings().weight.shape[0])
(venv_ji) root@jirack2:# python -c ' from transformers import AutoTokenizer tok = AutoTokenizer.from_pretrained("./QwenRoboticsTokenizer") print("Vocab size:", len(tok)) print("pad_token_id:", tok.pad_token_id) print("eos_token_id:", tok.eos_token_id) '
Vocab size: 151778 pad_token_id: 151643 eos_token_id: 151645
For joint ventures, hardware integration, or licensing inquiries:
MIT License