Qwen3-1.7B-CBPE: Qwen3-1.7B-Base on a Combinatorial BPE tokenizer

Qwen3-1.7B-Base with its 151k-row vocabulary replaced by a 131k-row unrestricted Combinatorial BPE vocabulary, then fully fine-tuned ("healed") on about 47B tokens. Every token is a tuple (variation, prefix, core, suffix) instead of one id, so the same word in its many spellings ( data, Data, DATA, data,...) shares one core row. The text needs about 1.6x fewer tokens (5.85 vs 3.61 bytes per token on the training mix) at a bits-per-byte cost of +3.1% on average (+2.4% weighted by bytes) against the original model on our test set (below).

What you get for that: the same text is about 1.5x shorter in tokens, so for the same token budget (and the same attention / KV-cache cost per token)

• about 1.5x more text fits in the context window (a 2,048-token window holds about 12k characters instead of about 7.4k), and

• about 1.5x more text is generated per second (measured, below), since every decoding step emits about 1.5x more text.

• Method, tokenizer trainer and code: https://github.com/SwayStar123/CombinatorialBPE

• This is a base model (no instruction tuning). The healing data contains Qwen-generated chat text in Qwen3's chat format, so it also follows that format loosely, but it is not a chat model.

• It is not a standard architecture: the vocabulary is factored, the embedding is a sum of four tables and the output is a four-step chain head. AutoModelForCausalLM / AutoTokenizer work through the custom code in this repository (trust_remote_code=True, below); stock transformers tooling that needs flat logits does not.

Use

pip install torch transformers safetensors huggingface_hub regex

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "SwayStar123/Qwen3-1.7B-CBPE"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)    # bf16 (about 3.5 GB), GPU if available
inputs = tok("The capital of France is", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=40, tokenizer=tok)             # tokenizer= turns token healing on
print(tok.decode(out[0]))

• input_ids are one integer per token: the four fields (variation, prefix, core, suffix) packed in mixed radix, so tok(...), padding, tok.decode(...) and model(input_ids, labels=input_ids).loss work as usual.
• The model has no flat vocabulary, hence no logits: a token is predicted field by field (chain head), so transformers pipelines, vLLM, llama.cpp and the like do not apply. model.generate is our own sampler (greedy, or do_sample=True with temperature / top_p / seed; stop_strings=[...] with tokenizer=) with a KV cache and token healing (the prompt's last token is dropped and the first new tokens must respell its bytes); model.token_nll(ids) gives the per-field NLL.
• No Rust needed. The tokenizer's encoder is a Rust program (native/dict_search in the GitHub repository); without it, a pure-Python encoder with identical output is used (it matched the Rust encoder on every chunk of ten test files, a mixed test set and 1.1M synthetic chunks; roughly 1-9k words per second). To use the fast one, build it and set DS_BIN=/path/to/dict_search.
• trust_remote_code=True runs the Python files in this repository (modeling_cbpe_qwen3.py, tokenization_cbpe_qwen3.py and what they import); read them first if you like, they are short. Tested with transformers 4.57 and 5.19, torch 2.13.

Without trust_remote_code, from a clone of https://github.com/SwayStar123/CombinatorialBPE (pip install -e ".[qwen3]"): from cbpe.qwen3 import load_model, generate; model, tok = load_model("SwayStar123/Qwen3-1.7B-CBPE"), or python -m cbpe.qwen3 --model SwayStar123/Qwen3-1.7B-CBPE --prompt "def fibonacci(n):". code/ in this repository is a snapshot of those files.

Architecture

The exact computation is in cbpe_qwen3.py (see the docstring); tests/test_qwen3.py checks it against the JAX training code.

Results

Bits per byte (lower is better) on out-of-distribution text, 2,048-token non-overlapping windows, about 200k characters per file (experiments/make_test.py of the jax-lm branch builds the files from Wikipedia, CodeSearchNet and CodeContests; the losses are the byte-exact attribution of experiments/attribution.py). The original Qwen3-1.7B-Base is scored with its own tokenizer, same 2,048-token context.

• Better on Hindi (-14%), level on German, Japanese and Java (-0% to +1%); worse on Go, JavaScript, English and Python (+3% to +8%), Chinese (+9%) and C++ (+15%).
• Fairness caveat: both models get 2,048 tokens of context, so this model sees about 1.6x more text in its window. Given the same number of bytes of context (a 1,264-token window for this model against 2,048 tokens for Qwen) the gap is +6.1% (mean of the ratios) / +4.7% (all bytes together).
• Where the loss is: common tokens are at parity; the remaining gap comes mostly from rare or unseen factored combinations (13% of the bytes, 63% of the gap), Chinese (+9%), capitalised words after a space, and C++ (+15%; the C++ test is competitive-programming code, which the healing data lacks).
• Speed (measured): greedy generation with Hugging Face eager generate on one RTX 3090, bf16, 8 code prompts (Python, C++, Rust, JavaScript, SQL), both models forced to generate full length: this model 38.9 ms per token and 6.05 bytes per token = 155 bytes/s; Qwen3-1.7B-Base 36.6 ms per token and 3.86 bytes per token = 105 bytes/s, i.e. about 1.5x more text per second (each step costs about 6% more here, the chain head adds about 2 ms, but emits 1.57x more text). Both are kernel-launch bound in eager mode (about 2,000 kernels per token), so absolute numbers will be much higher with CUDA graphs or torch.compile; the ratio should hold. Training throughput was also equal in tokens/s (223.7k vs 220.1k on 2 nodes). Tokenisation: the pure-Python encoder does roughly 1-9k words per second (the Rust one, used if built, is roughly 10-100x faster).

Example: code completion, same prompt, greedy

Prompt (13 tokens for this model, 16 for Qwen3-1.7B-Base):

def is_prime(n):
    """Return True if n is a prime number."""

This model (continuation):

    if n <= 1:
        return False
    if n <= 3:
        return True
    if n % 2 == 0 or n % 3 == 0:
        return False
    i = 5
    while i * i <= n:
        if n % i == 0 or n % (i + 2) == 0:
            return False
        i += 6
    return True

Qwen3-1.7B-Base (continuation):

    if n < 2:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True

Both are correct; this one uses the 6k +/- 1 optimisation. The completions of the other prompts (merge sort, an HTTP fetch with retries, an LRU cache, binary search in C++, a Rust word count, SQL, a JavaScript debounce) are of similar quality, with some weaker spots: the Rust sketch forgets its use lines, and this model never emits an end-of-text token, so it keeps writing the next function or explanation after finishing; set max_new_tokens or stop_strings.

Checks against the training code

The PyTorch port was compared with the JAX training code (jax-lm branch) on these exact weights:

• Tiny random model, fp32: per-token, per-field NLL equal to the JAX run within 2e-4 (tests/test_qwen3.py).
• Real weights, bf16: one 2,048-token window of each of the 10 test files, tokens encoded by this repository's dict_search (identical token ids to the trainer's). The mean NLL per token differs from JAX by 0.00-0.003 nats on every file (en 2.9202 vs 2.9206, python 2.2852 vs 2.2858, zh 3.8191 vs 3.8180), i.e. below 0.1%, with no systematic sign; single tokens differ by about 0.02 nats on average (bf16 arithmetic differs: JAX keeps the residual stream in fp32, transformers' Qwen3 in bf16).
• The bf16 export itself costs nothing: bits per byte of the fp32 training weights and of the bf16 weights agree to the fourth decimal on all 10 files (JAX, whole files).
• Greedy generation vs JAX (4 prompts, 40 tokens, healing on): two identical for all 40 tokens, one for 25 tokens (then a near-tie goes the other way); the fourth differs from the second token on, because token healing picks among sampled candidates (different random draws in JAX and PyTorch).

Training

Starting from the Qwen3-1.7B-Base weights, with new tables (initialised so that the sum of a token's four rows approximates the mean of Qwen's embeddings of the token's text) and head, trained with AdamW: 45,000 steps x 512 sequences x 2,048 tokens (about 47B tokens, 270 GB of text), learning rate 5e-5 (cosine to 10%; 1e-3 for the new tables and head, which alone were trained for the first 500 steps), on a mix of:

• FineWeb-Edu 1.3.0 (sample-100BT): English, the largest share
• FinePDFs: German, French, Hindi, Japanese
• FineWeb-2: Mandarin (cmn_Hani)
• common-pile Comma v0.1 stackv2_edu: code
• Qwen-generated chat data: Magpie-Qwen2.5-Pro-1M, Magpie-Qwen2-Pro-1M, and a-m-team/AM-Qwen3-Distilled (all six files: other, multiturn, if, science, code, math). AM-Qwen3-Distilled states no licence on its dataset card.

The chat data was rendered in Qwen3's chat format as plain text. Licence notes: the base model, Qwen3-1.7B-Base, is Apache-2.0, and this model is released under the same licence. The healing data comes from the datasets above, each under its own terms (see their cards); in particular the chat data is text generated by Qwen models, and AM-Qwen3-Distilled declares no licence at all. Check them if that matters for your use. The code (code/ and the Python files in the repository root) is MIT.

Files

• model.safetensors: all weights (bf16). Body tensors use Hugging Face Qwen3 names (model.layers.N...), plus emb.0-3.weight (the four tables), steps.K.* and ln_steps.K.* (the chain head).
• config.json: the Qwen3 config plus auto_map and cbpe (table sizes, field order, tokenizer path).
• tokenizer/cbpe_tokenizer.json and .bin: the tokenizer; tokenizer_config.json.
• modeling_cbpe_qwen3.py, configuration_cbpe_qwen3.py, tokenization_cbpe_qwen3.py, cbpe_qwen3.py, unrestricted.py, unrestricted_py.py: the remote code (the model, the chain head and sampler; the tokenizer and its pure-Python encoder).
• code/: snapshot of the GitHub repository files for the clone-based route (package, Rust encoder source, tests, docs).
• results_bpb_bf16_vs_fp32_jax.json, parity_nll_torch_vs_jax.json: the numbers behind the parity checks; results_attribution_j300.json: the per-file bits behind the results table.