GPT-Compact
GPT-Compact is a text generation model meant to answer the question: "How strong of a model can Google Colab create?"
Model Description
GPT-Compact has 16 heads, 24 layers, and 1280 embeddings, similar to GPT2-Large, with a context window of 16,384 tokens. This maximized GPU VRAM while balancing size and functionality. It was trained on a diverse corpus of over 17 billion tokens.
Model Architecture
| Layers | 24 |
| Heads | 16 |
| Embeddings | 1280 |
| Context Window | 16384 tokens |
| Tokenizer | cl100k_base |