Atto
Atto is a family of compact contextual models designed for parameter-efficient language representation learning.
These models can be loaded using Hugging Face Transformers. Since Atto uses a custom architecture, trust_remote_code=True is required.
from transformers import AutoModel
model = AutoModel.from_pretrained(
"neuml/atto-9k-16d",
trust_remote_code=True
)
Training Parameters
| Parameter | Value |
|---|---|
| learning_rate | 0.001 |
| gradient_accumulation_steps | 1 |
| fp16 | True |
More Information
See the Introducing Atto article for benchmark results, architecture details, and training methodology.