Model Card for amusktweewt/tiny-stories-v1

This model is a custom transformer-based language model trained on the TinyStories dataset, designed for creative text generation tasks such as storytelling and conversational agents. This model is purely an academic project and should not be used in production or practical applications.

Model Details

Model Description

This model utilizes a custom tokenizer with Byte Pair Encoding (BPE) and has been trained with a smaller architecture to balance efficiency and performance. It is designed for generating coherent and contextually relevant short stories. However, a known issue with the tokenizer causes spaces between tokens to appear repeated, leading to suboptimal text output quality.

• Developed by: amusktweewt
• Model type: AutoModelForCausalLM
• Language(s) (NLP): English
• License: MIT

Model Sources

• Repository: HuggingFace repository

Uses

Direct Use

This model is intended for academic and research purposes only. It demonstrates a proof of concept for training smaller transformer-based language models.

Out-of-Scope Use

• Not suitable for tasks requiring factual accuracy
• Should not be used in production environments or applications involving sensitive content

Bias, Risks, and Limitations

Risks and Biases

The model may reflect biases present in the training data, leading to unintended or inappropriate outputs. Additionally, the tokenizer issue can result in suboptimal and incoherent text generations.

Recommendations

This model is meant for research and demonstration purposes. Users should validate outputs critically and avoid using it for practical applications.

How to Get Started with the Model

from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

model = AutoModelForCausalLM.from_pretrained("amusktweewt/tiny-stories-v1")
tokenizer = PreTrainedTokenizerFast.from_pretrained("amusktweewt/tiny-stories-v1")

prompt = "Once upon a time,"
inputs = tokenizer(prompt, return_tensors="pt", return_token_type_ids=False)
outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

The model was trained on the TinyStories dataset, consisting of curated short stories. Preprocessing ensured consistent formatting and tokenization using a custom BPE tokenizer.

Training Procedure

Preprocessing

• Used BPE tokenizer with a vocabulary size of 4096
• Included special tokens: <sos>, <pad>, <|endoftext|>, and <unk>

Training Hyperparameters

• Batch size: 64
• Epochs: 3
• Learning rate: 1e-3
• Scheduler: Cosine annealing
• Precision: Mixed precision (FP16)

Speeds, Sizes, Times

• Training time: Approx. 5 hours 30 minutes
• Model size: 230 MB
• Dataset size: 535.98 million tokens

Evaluation

Testing Data, Factors & Metrics

Testing Data

A subset of the training data was used for evaluation, focusing on coherence and storytelling quality.

Metrics

• Loss: 0.9723
• Qualitative Evaluation: Manual assessment of generated outputs for coherence and relevance.

Results

• Sample Outputs:
  • Prompt: "in a far away country" Completion: "in a far away coun try . He was so excited to explore the world . He was so happy to be able to explore the world ."

Summary

The model generates coherent short stories suitable for research demonstration but is limited by tokenizer issues and should not be used in real-world scenarios.

Environmental Impact

• Hardware Type: NVIDIA 4090 GPU
• Hours used: 5.5
• Carbon Emitted: Approx. 0.2 kg CO2 eq

Technical Specifications

Model Architecture and Objective

• Transformer architecture with 8 layers, 12 attention heads, and a hidden size of 768

Compute Infrastructure

Hardware

• Single GPU (NVIDIA 4090)

Software

• Python 3.8+
• HuggingFace Transformers 4.x
• PyTorch 1.x

Model Card Authors

amusktweewt

Model Card Contact

For questions or feedback, contact amusktweewt.