Atto is a family of compact contextual models designed for parameter-efficient language representation learning.
This is a model fined-tuned using sentence-transformers.
This model can be used to build embeddings databases with txtai for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).
Note: txtai 9.0+ is required for sparse vector scoring support
import txtai
model = "atto-17m-8192d-sparse-embeddings"
embeddings = txtai.Embeddings(
sparse=model,
content=True
)
embeddings.index(documents())
# Run a query
embeddings.search("query to run")
Alternatively, the model can be loaded with sentence-transformers.
from sentence_transformers import SparseEncoder
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SparseEncoder(model)
embeddings = model.encode(sentences)
print(embeddings)
The following table shows a subset of BEIR scored with the txtai benchmarks script. Scores reported are ndcg@10.
| Model | Parameters | Dimensions | Vocab | ArguAna | NFCorpus | SciFact | Average |
|---|---|---|---|---|---|---|---|
| atto-17m-8192d-sparse | 17M | 8192 | 30K | 0.3981 | 0.3208 | 0.6600 | 0.4596 |
| atto-28m-8192d-sparse | 28M | 8192 | 30K | 0.3651 | 0.3292 | 0.6618 | 0.4520 |
| SlideOn | 174M | 768 | 50K | 0.4655 | 0.3863 | 0.7529 | 0.5349 |
| Parameter | Value |
|---|---|
| per_device_train_batch_size | 64 |
| num_train_epochs | 25 |
| learning_rate | 0.0003 |
| warmup_steps | 1000 |
| weight_decay | 0.01 |
| gradient_accumulation_steps | 2 |
| fp16 | True |
| per_device_eval_batch_size | 64 |
| dataloader_num_workers | 20 |
| prompts | {'query': 'query: ', 'document': 'document: '} |
SparseEncoder(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'AttoModel'})
(1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 8192})
)
See the Introducing Atto article for benchmark results, architecture details, and training methodology.
NeuML/atto-17m-8192d