GODELEV TOK-4K V2 Production Tokenizer

This repository contains the official weights and configuration for the GODELEV TOK-4K tokenizer, a highly optimized, custom Byte-Pair Encoding (BPE) tokenizer engineered for hybrid text, mathematical reasoning, and structured chain-of-thought processing.

Designed with a compact, ultra-dense vocabulary layout, this tokenizer enforces strict token boundaries to eliminate common serialization errors seen in flagship model architectures.

---

Technical Specifications

---

Architectural Approach and Design Rationale

Compact Vocabulary Strategy

The selection of a 4,096 vocabulary size represents an explicit architectural trade-off engineered for specific sequence modeling environments. While standard downstream Large Language Models (LLMs) often scale vocabularies to 32,000, 50,000, or over 100,000 tokens to maximize sequence compression, a 4,096 vocabulary size drastically minimizes the embedding and final language model head layer parameters. This configuration ensures that small-scale or highly specialized model architectures allocate their capacity entirely to hidden layer representations and deep sequence dependencies rather than wide sparse matrix lookup tables.

Structural Integrity via tokenizer_config.json

The underlying layout serialization in tokenizer_config.json enforces several core programmatic constraints to ensure robust text-to-ID and ID-to-text cycles:

1. Explicit Token Protection: All control and special tokens are registered with the parameter "normalized": false. This prevents the Unicode Normalization Form KC (NFKC) from altering strings that resemble special structures. For instance, even if a text input contains unusual white space or canonical variants, it will never break the underlying <|think|> or <|user|> strings into sub-word byte patterns.
2. Deterministic Control ID Mapping: Special system tokens occupy indices 0 through 16. This continuous block guarantees that control characters reside safely away from raw byte representations and dynamically learned BPE merge structures, simplifying mask calculation and vocabulary slicing inside transformer custom heads.
3. Disabled Space Cleanup: The flag "clean_up_tokenization_spaces": false is explicitly set. Standard tokenizers frequently strip continuous spaces, leading to corruption in code syntax formatting or aligned tabular mathematical expressions. Preserving whitespaces directly supports structured chain-of-thought processing.

Custom Pre-Tokenizer Pipeline

To prevent structural degradation across text transitions, the tokenizer executes a strict sequence pipeline:

1. Isolated Splitting: Uses a specialized regex engine pattern identical to advanced modern architectures:

   '(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]+|(?:\s)[\s]+|[\p{L}]+|[\p{N}]|[\r\n]
   

   This splits inputs cleanly into explicit word stems, contractions, sequential punctuation groupings, numerical digits, and newline flags before the BPE algorithms calculate merge metrics.
2. Byte-Level Fallback: Maps the split substrings into byte fragments. This bypasses the typical constraints of closed-vocabulary character maps, completely mitigating Out-of-Vocabulary (OOV) tokens. Any unknown character sequences or arbitrary binary inputs are safely broken down into basic native bytes without crashing runtime workflows.

---

Dataset Training Composition

The tokenizer was trained on a rich, multi-domain corpus containing 1,113,659 unique documents comprising 4,186,212,111 total characters (approximately 4.18 GB of dense training data). The corpus combines massive scale conversational reasoning with high-fidelity exact mathematical training instances:

GODELEV/BetterDataset-2M

• Data Volume: 500,000 items extracted natively via stream buffers.
• Targeted Fields: ["text"]
• Purpose: Provides rich linguistic variety, comprehensive semantic vocabulary patterns, conversational grammar frameworks, and broad real-world knowledge distributions.

GODELEV/Arithmetic

• Data Volume: 96,103 static rows (100% extraction of the full curriculum dataset).
• Targeted Fields: ["text"]
• Purpose: Instills fine-grained algorithmic sequences, number operations, math equations, and continuous text patterns optimized for logical expression evaluations without experiencing numeric digit splitting artifacts.

---

Special Token Registry

Special and additional structural tokens are embedded at the lowest vocabulary indexes to preserve strict interaction parameters:

---

Usage and Implementation

To initialize this tokenizer from the Hugging Face Hub, ensure you have the transformers and tokenizers libraries installed, then instantiate via AutoTokenizer:

from transformers import AutoTokenizer

# Load the production tokenizer
tokenizer = AutoTokenizer.from_pretrained("GODELEV/TOK-4K")

# Example sample showcasing structured chain-of-thought integration
sample_phrase = "<|user|>\nExecute calculation <|think|> Processing <|end_of_think|> <|math|>2+2<|answer|>4"

# Encode to sequence IDs
encoded_ids = tokenizer.encode(sample_phrase)
print(f"Encoded IDs: {encoded_ids}")

# Round-trip decoding
decoded_string = tokenizer.decode(encoded_ids)
print(f"Decoded Output: {decoded_string}")

---

Maintenance and Preservation

This tokenizer artifact ensures zero index collisions for downstream execution tasks. When moving weights into active model pipelines, do not call native text normalization routines over the string payloads manually, as this might bypass the internal safety parameters mapped into the AddedToken protections.