Dhara-70M

A 70M parameter diffusion language model optimized for high-throughput text generation with superior factuality.

Table of Contents

• Model Description
• Training Data
• Training Details
• Benchmark Results
• Usage
• Key Insights
• Limitations
• Citation

Model Description

Dhara-70M is a novel diffusion language model that achieves:

• 3.8x higher throughput than autoregressive models
• Best-in-class factuality on TruthfulQA (47.50%)
• 10x training efficiency via WSD (Warmup-Stable-Decay) conversion

Architecture

Training Data

Dhara was trained in two stages:

Stage 1: AR Pretraining (1B tokens)

• 40% FinePDFs (400M tokens)
• 30% DCLM Baseline (300M tokens)
• 30% FineWeb-Edu (300M tokens)

Stage 2: WSD Conversion (100M tokens)

• Progressive block size warmup (1→4→32→64→1024)
• MDLM diffusion objective

Training Details

Benchmark Results

Inference Performance

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("codelion/dhara-70m")
model = AutoModelForCausalLM.from_pretrained(
    "codelion/dhara-70m",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16
)

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

# Generate text
prompt = "The future of artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
outputs = model.generate(
    inputs.input_ids,
    max_new_tokens=50,
    temperature=0.1,
    top_p=0.5,
    top_k=5,
    repetition_penalty=1.8,
    do_sample=True,
    pad_token_id=0
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Example Output:

The future of artificial intelligence is a big challenge.
This world has the potential to improve, but this time we have no other than "theworld."
The next generation will be more exciting and its very much important for our society's
abilityto develop its

Batch Generation (High Throughput)

# For batch generation, use larger batch sizes
prompts = [
    "The future of artificial intelligence is",
    "The human brain is capable of",
    "Science has shown that",
    "Technology continues to evolve"
]

inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(device)
outputs = model.generate(
    inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=50,
    temperature=0.1,
    top_p=0.5,
    top_k=5,
    repetition_penalty=1.8,
    do_sample=True,
    pad_token_id=0
)

for i, output in enumerate(outputs):
    print(f"Output {i+1}: {tokenizer.decode(output, skip_special_tokens=True)}")

Key Insights

1. Throughput vs Accuracy Trade-off: Dhara trades 1.33% average accuracy for 3.8x higher throughput, making it ideal for batch processing tasks.

2. Superior Factuality: Dhara excels on TruthfulQA (+1.67% vs GPT-2), suggesting diffusion models may reduce hallucinations through bidirectional context.

3. Reasoning Advantage: ARC-Challenge +2.56% indicates strong performance on reasoning tasks.

4. WSD Efficiency: Converting an AR model to diffusion via WSD uses 10x fewer tokens than training from scratch with equivalent quality.

5. Canon Layers Help: The depthwise causal convolutions (Canon layers) improve factuality and reasoning with only 0.13% parameter overhead.

Limitations

• Lower performance on sequential reasoning tasks (GSM8K: 0.00%)
• Higher memory usage due to bidirectional attention
• Slightly higher time-to-first-token latency
• Best suited for batch rather than interactive use cases

Citation

@article{sharma2025optimal,
  title={The Optimal Architecture for Small Language Models},
  author={Sharma, Asankhaya},
  year={2025},
  url={https://huggingface.co/blog/codelion/optimal-model-architecture}
}

Related Work

• The Optimal Architecture for Small Language Models - Blog post describing this work
• The 1 Billion Token Challenge: Optimal Dataset Mixing - Our previous work on optimal pretraining data
• GPT-2-70M - Our previous model from optimal pretraining experiments

Contact

For questions or feedback, please open a discussion on the Hugging Face discussions page.