TinyStories GPT (30M Parameters)

Custom 30M parameter GPT model trained from scratch on the TinyStories dataset.

Model Details

• Architecture: Decoder-only Autoregressive Transformer (GPT)
• Parameters: 120.14M
• Context Window (block_size): 512 subword tokens
• Embedding Dimension (n_embd): 384
• Layers / Heads: 6 layers, 6 heads
• Tokenizer: BPE (tiktoken GPT-2 vocabulary)
• Weight Tying: Enabled
• Precision: 16-bit mixed precision (bfloat16/float16)

How to Use

from transformers import GPT2Tokenizer, pipeline

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
generator = pipeline("text-generation", model="Bwenge840/tinystories-gpt-30m", tokenizer=tokenizer)

story = generator("Once upon a time, a little girl named Jessica", max_new_tokens=150, do_sample=True, top_k=40, temperature=0.8)
print(story[0]["generated_text"])