A custom GPT-style language model trained from scratch using PyTorch.
| Parameter | Value |
|---|---|
| Architecture | GPT (Decoder-only Transformer) |
Hidden size (d_model) |
768 |
| Attention heads | 8 |
| Transformer blocks | 1 |
| Max sequence length | 1024 |
| Vocabulary size | 32000 |
| Dropout | 0.2 |
Custom BPE tokenizer trained with the HuggingFace tokenizers library.
Special tokens: <|endoftext|> · <|pad|> · <|unk|>
You can easily load this model and tokenizer using the transformers library. Because the model uses a custom architecture, you must pass trust_remote_code=True.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("shubhamprakash108/decoder-only-transformer-tinystories-32m", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("shubhamprakash108/decoder-only-transformer-tinystories-32m", trust_remote_code=True)
# Set up device
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
# Generate text
prompt = "The transformer is based on"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
output_ids = model.generate(
**inputs,
max_new_tokens=50,
do_sample=True,
temperature=0.8,
pad_token_id=tokenizer.eos_token_id
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
Apache 2.0