Tri-70B-preview-SFT

Introduction

We introduce Tri-70B-preview-SFT, a research preview of our latest and largest flagship language model that redefines the efficiency frontier in LLM training. By achieving frontier performance for it's compute size (1.5T training tokens from scratch), we demonstrate that exceptional capabilities don't require excessive computational resources.

We are releasing a minimally post-trained version to enable open research and community experimentation. This preview version has only undergone supervised fine-tuning and has not been subjected to extensive RLHF. This enables researchers to explore various RL-based alignment techniques with this model. Stay tuned for the base model release coming soon!

Key Highlights

• Architecture optimized for long context
  • 32k context window
  • Sliding window attention with window size 4096
  • iRoPE: Interleaved local (RoPE) and global (temperature-scaled) attention
  • Scalable softmax
• Multi-lingual capabilities: Specially optimized for English, Korean, and Japanese
• Enhanced reasoning: Modified training dataset mixture specifically designed for reasoning capabilities, with emphasis on step-by-step problem solving
• Minimal post-training: This preview release features only supervised fine-tuning, enabling researchers to explore custom alignment techniques and RLHF/RLVR approaches

Model Specifications

Tri-70B-preview-SFT

Quickstart

Here is a code snippet with apply_chat_template that demonstrates how to load the tokenizer and model and generate text:

Tri-70B-SFT Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "trillionlabs/Tri-70B-preview-SFT"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "Explain the concept of central limit theorem in simple terms."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

vLLM Deployment

We are working with vLLM team on native vLLM support of our Trillion model. In the meantime, we offer an off-the-tree integration method (link).

# install off-the-tree model integration plugin
git clone https://github.com/trillion-labs/trillion-deployment-utils.git
cd trillion-deployment-utils && pip install -e .

vllm serve trillionlabs/Tri-70B-preview-SFT --trust-remote-code

The plugin has been tested on vLLM version 0.10.1.1. Check out the repository for detailed usages.

Evaluation

We evaluated Tri-70B-preview-SFT across a suite of benchmarks assessing general reasoning, knowledge recall, coding abilities, mathematical reasoning, and instruction-following capabilities. We compare our model against state-of-the-art models of similar scale: Qwen-2.5-72B-instruct and Llama-3.1-70B.

Full evaluation settings

Benchmark Evaluation Settings

**Note that MT-Bench uses a 10-point scale.

Benchmark Results

Models compared:

• Tri-70B-preview-SFT: Our flagship 70B parameter model
• Qwen-2.5-72B-instruct: Qwen's 72B parameter instruction-tuned model
• Llama-3.1-70B: Meta's instruction-tuned 70B model

Limitations

• Language Support: The model is optimized for English, Korean, and Japanese. Usage with other languages may result in degraded performance.
• Knowledge Cutoff: The model's knowledge is limited to information available up to February 2025.
• Minimal Post-Training: As this is a supervised fine-tuning (SFT) release without RLHF, responses may occasionally lack the polish and safety alignment of fully post-trained models.

License

This model repository is licensed under the Trillion License.

Contact

For inquiries, please contact: info@trillionlabs.co