GPT-Usenet-6
A 0.4B parameter LLM using GPT-4 encodings. Trained using 30GB of USENET posts, web scrapes, and stories. Supervised fine-tuning should be performed before use.
Technical Information
| Layers | 24 |
| Heads | 16 |
| Embeddings | 1024 |
| Context Window | 16384 tokens |
| Tokenizer | cl100k_base |
Training Information
| Training Loss | 2.5916 |
| Validation Loss | 2.7335 |
| Device | Google Colab G4 |
| Training Time | 24 Hours |