HRM-Text-0.6B

HRM-Text-0.6B is a dense HRM-Text L-size base checkpoint trained with the HRM pretraining pipeline. This release is the epoch-4 pretrained checkpoint, exported from the native FSDP2 checkpoint into a single bf16 model.safetensors file.

The model has 694.7M dense parameters, so every token activates the full 0.6B parameter budget. It is the dense active-compute baseline for the active-matched L-MoE 64x8 experiment in Xiaoye08/HRM-MoE.

This is a pre-alignment base checkpoint. It is not a chat or instruction-following assistant.

Model Structure Comparison

HRM-Text-0.6B dense L

input tokens
  -> H/L recurrent HRM blocks
  -> dense SwiGLU FFN, intermediate width 3584
  -> all dense FFN parameters are active for every token

Active-matched L-MoE 64x8

input tokens
  -> H/L recurrent HRM blocks
  -> router over 64 SwiGLU experts
  -> top-8 experts active per token, each expert width 448
  -> active FFN width = 8 x 448 = 3584

The MoE variant matches the active FFN width of this dense L checkpoint while increasing total expert capacity.

Results vs Active-Matched L-MoE

The comparison below uses the same sampled pretraining data, same 32-GPU setup, and 4 pretraining epochs. Values are percentages unless otherwise noted.

This dense checkpoint is the main 0.6B baseline. The active-matched L-MoE improves Standard and MMLU-Pro metrics while leaving AIME25 majority voting roughly unchanged.

Model Details

Requirements

This checkpoint includes remote code for the hrm_text architecture so it can be loaded on Transformers versions that do not yet ship native HRM-Text support.

pip install --upgrade transformers safetensors accelerate

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Xiaoye08/HRM-Text-0.6B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).cuda().eval()

# synth,cot composite: reasoning / CoT style.
condition = "<|quad_end|><|object_ref_end|>"
prompt = f"<|im_start|>{condition}Explain why the sky is blue.<|im_end|>"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
inputs["token_type_ids"] = torch.ones_like(inputs["input_ids"])

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=False))

Prompt Format

HRM-Text uses condition prefix tokens. Prompts should be rendered as:

<|im_start|><condition tokens>prompt text<|im_end|>

Common conditions:

• direct -> <|object_ref_start|>
• cot -> <|object_ref_end|>
• noisy -> <|quad_start|>
• synth -> <|quad_end|>

For reasoning-style prompting, synth,cot maps to <|quad_end|><|object_ref_end|>.

PrefixLM Mask

The checkpoint was pretrained with the HRM-Text PrefixLM objective. Pass token_type_ids = torch.ones_like(input_ids) for the prompt, marking it as one bidirectional prefix block before autoregressive decoding.

Training Snapshot

• 32 GPUs
• 4 pretraining epochs
• global batch size 172,032 tokens
• learning rate 2.5e-4
• bfloat16 forward/backward
• EMA decay 0.9999
• official HRM-Text L configuration

Limitations

• Pre-alignment base checkpoint, not a chat model.
• Not instruction-tuned, RLHF-trained, or safety-aligned.
• English-focused pretraining mixture.
• Requires trust_remote_code=True unless using a Transformers version with native hrm_text support.
• Outputs may be inaccurate, biased, or unsafe.

License

Apache License 2.0.

Citation

This model is derived from HRM-Text. If you use HRM-Text, please cite:

@misc{wang2026hrmtextefficientpretrainingscaling,
      title={HRM-Text: Efficient Pretraining Beyond Scaling},
      author={Guan Wang and Changling Liu and Chenyu Wang and Cai Zhou and Yuhao Sun and Yifei Wu and Shuai Zhen and Luca Scimeca and Yasin Abbasi Yadkori},
      year={2026},
      eprint={2605.20613},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.20613},
}

Related Models

• HRM-MoE: https://huggingface.co/Xiaoye08/HRM-MoE
• HRM-Text reference model: https://huggingface.co/sapientinc/HRM-Text-1B
• HRM-MoE code: https://github.com/XiaoYee/HRM-MoE