Ettinnet Preview
Ettinnet Preview is a series of Ettin Encoder models finetuned for fill-masking using messages sourced from sites such as Discord, Reddit, and Twitter for better understanding of internet slang.
Training
Ettinnet was trained on 42 million samples of selected messages from the ManBib/DUE-filtered-deduped dataset.
Additionally, ~750,000 samples of text were added from HuggingFaceFW/fineweb, HuggingFaceFW/fineweb-edu, and wikipedia in order to reduce catastrophic forgetting.
Future iterations of the model will be trained on additional datasets including Reddit and Twitter messages/comments.
Benchmarks
The model was evaluated on a subset of 100,000 random Discord messages with 182k masked tokens from the ManBib/DUE-filtered-deduped dataset excluded its training data.
| Metric | ettin-encoder-400m | ettinnet-17m-preview | ettinnet-32m-preview |
|---|---|---|---|
| Top-1 | 34.57% | 34.27% | 37.79% |
| Top-3 | 47.73% | 48.89% | 52.74% |
| Top-5 | 52.59% | 54.60% | 58.28% |
| Top-10 | 58.32% | 61.60% | 65.04% |
| Top-20 | 63.31% | 67.87% | 70.94% |
| MRR | 0.4298 | 0.4382 | 0.4737 |
| Cross-entropy | 4.3792 | 3.8370 | 3.5506 |
| Perplexity | 79.77 | 46.39 | 34.83 |
Top-1 by message length
| Words | Masked tokens | ettin-encoder-400m | ettinnet-17m-preview | ettinnet-32m-preview |
|---|---|---|---|---|
| 3–5 | 30,574 | 13.15% | 22.40% | 24.45% |
| 6–10 | 57,835 | 28.99% | 33.40% | 36.54% |
| 11–20 | 46,338 | 40.85% | 37.82% | 41.75% |
| 21–40 | 30,336 | 47.07% | 39.16% | 43.33% |
| 41–80 | 13,489 | 49.95% | 39.48% | 44.50% |
| 81–200 | 5,470 | 52.87% | 39.73% | 44.73% |
Limitations
- These are experimental preview models and not intended for production environments
- The models suffer at higher contexts due to limited training data of longer messages
- The model may forget coding and math and have a lower performance on them