hunmin-13B-A2B-v0.11-base

13B total / ~2B active parameter Mixture-of-Experts (MoE) language model with custom GptOssForCausalLM architecture.

Model Architecture

Tokenizer

Important: SpmDirectTokenizer does NOT pre-split on whitespace (unlike LlamaTokenizer). Text is passed directly to SentencePiece, preserving native BPE merges including the ▁ space prefix. The tokenizer code is in training/spm_tokenizer.py.

Usage

from spm_tokenizer import SpmDirectTokenizer

tokenizer = SpmDirectTokenizer(vocab_file="tokenizer.model", bos_token="<s>")
ids = tokenizer.encode("Hello world")

Note: Do NOT use AutoTokenizer with this model — it will silently load a HuggingFace fast tokenizer that pre-splits on whitespace, producing different tokenization.

Training History

Phase 1: 4K Context Pre-training

Phase 2: 16K Context Extension

Phase 3: 16K Tokenizer Remap

Phase 4: 64K Context Extension (In Progress)

Data

• Dataset: FineWeb (parquet shards, streaming mode)
• Path: text_corpora_fineweb_full_run02
• Total Shards: 387,270
• Rows per Shard: 10,000
• Consumed Shards (as of Phase 3 end): 3,211 / 387,270

Data State (for resuming)

The data_state.json files track which shards have been consumed. To resume training from where it left off:

{
  "global_step": 2000,
  "consumed_samples": 32110000,
  "consumed_shards": 3211,
  "rows_per_shard": 10000,
  "seed": 42
}

Pass this via --resume_data_state <path_to_data_state.json> to avoid re-training on already consumed data.

Training Code

All training code is in the training/ directory:

How to Resume Training

1. Use the latest checkpoint as --resume_from_checkpoint
2. Use the corresponding data_state.json as --resume_data_state to skip consumed shards
3. Ensure spm_tokenizer.py is in the same directory as train.py
4. Key arguments:

   torchrun --nnodes=5 --nproc_per_node=8 ... train.py \
     --model_config_json moe_design_hunmin-fm-13B-A2B.json \
     --tokenizer_name_or_path <model_dir_with_tokenizer.model> \
     --train_path <parquet_data_dir> \
     --streaming --packing \
     --resume_from_checkpoint <checkpoint_dir> \
     --resume_data_state <data_state.json> \
     --attn_implementation sdpa \
     --max_length <context_length> \
     --per_device_train_batch_size <bs> \
     --gradient_accumulation_steps 1 \
     --gradient_checkpointing \
     --fsdp \
     --save_only_model
   

Environment