Llama-3.1-8B Diffusion Model (LAD)

This is a Language Autoregressive Diffusion (LAD) model based on Llama-3.1-8B-Instruct.

Features

• 🎯 Dual mode: Autoregressive + Diffusion generation
• 🚀 Cosine noise schedule with 1000 timesteps
• 🧠 LoRA fine-tuning (rank 32)
• ⚡ Custom diffusion components

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("rootxhacker/llama3-diffusion")
tokenizer = AutoTokenizer.from_pretrained("rootxhacker/llama3-diffusion")

# Generate text
inputs = tokenizer("The future of AI", return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0]))

Training Details

• Base: Meta-Llama-3.1-8B-Instruct
• Dataset: PatrickHaller/cosmopedia-v2-1B
• Framework: Unsloth + Custom Diffusion
• Context: 256 tokens
• Training: 60% AR + 40% Diffusion

Uploaded: 2025-06-08 23:13