ettin-32m-distil-gte

This is a sentence-transformers model finetuned from capemox/ettin-encoder-32m-pretrain-50m using the pooled GTE-ModernBERT embeddings published in the GTE modernbert Embedpress collection: MS MARCO queries, MS MARCO documents, PubMedQA queries, GooAQ queries, and SQuAD queries. It maps inputs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: capemox/ettin-encoder-32m-pretrain-50m
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Datasets:
  • stephantulkens/msmarco-query-gte-modernbert-pooled
  • stephantulkens/msmarco-gte-modernbert-pooled
  • stephantulkens/pubmedqa-query-gte-modernbert-pooled
  • stephantulkens/gooaq-query-gte-modernbert-pooled
  • stephantulkens/squad-query-gte-modernbert-pooled
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 384, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("much1na/ettin-32m-distil-gte")

# Run inference with queries and documents
query = "Which planet is known as the Red Planet?"
documents = [
    "Venus is often called Earth's twin because of its similar size and proximity.",
    "Mars, known for its reddish appearance, is often referred to as the Red Planet.",
    "Jupiter, the largest planet in our solar system, has a prominent red spot.",
    "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
]
query_embeddings = model.encode_query(query)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# (768,) (4, 768)

# Compute similarities to determine a ranking
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7675, 0.8726, 0.8469, 0.8348]])

Evaluation

Metrics

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
    "dataset_names": [
      "climatefever",
      "dbpedia",
      "fever",
      "fiqa2018",
      "hotpotqa",
      "msmarco",
      "nfcorpus",
      "nq",
      "quoraretrieval",
      "scidocs",
      "arguana",
      "scifact",
      "touche2020"
    ],
    "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Baselines

Mean cosine NDCG@10 over the 13 NanoBEIR datasets. The CSV files are in eval/nanobeir/.

The 128-token teacher and untrained student runs were repeated on Colab in one job. The 256-token teacher and final model results come from earlier runs with the same evaluator.

Training Details

Baselines

Before training, the notebook evaluated the teacher and the untrained student on the full NanoBEIR benchmark (mean cosine NDCG@10):

The teacher and student baselines used a maximum sequence length of 128. The training run used 256 tokens.

Training Datasets

The links below point to the public datasets in the GTE modernbert Embedpress collection. The collection contains the complete embedded corpora; this run used the subsets shown below (5,870,041 training examples in total). The MS MARCO document subset used 4,000,000 rows (3,999,008 remained after filtering texts over 256 tokens), and the GooAQ subset used 500,000 rows. The other datasets were used in full.

msmarco-query-gte-modernbert-pooled

msmarco-query-gte-modernbert-pooled

• Dataset: stephantulkens/msmarco-query-gte-modernbert-pooled
• Size: 1,010,916 training samples
• Columns: text and label
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	min: 5 tokens mean: 9.97 tokens max: 19 tokens	size: 768 elements

• Samples:

  text	label
  is pneumococcal polysaccharide vaccine a live vaccine	[0.22062334418296814, -0.47287487983703613, -1.071263074874878, -1.1376807689666748, 1.1372895240783691, ...]
  adding a picture to skype profile	[1.0464050769805908, -1.0763801336288452, 1.5920714139938354, -0.6542222499847412, -0.048727527260780334, ...]
  how long are you contagious with flu	[1.2045410871505737, 1.0756381750106812, 2.6750006675720215, -0.15280567109584808, 1.2226042747497559, ...]

• Loss: EmbedDistillLoss with these parameters:

  {
    "distance_metric": "cosine",
    "projection_dim": null
  }
  

msmarco-gte-modernbert-pooled

msmarco-gte-modernbert-pooled

• Dataset: stephantulkens/msmarco-gte-modernbert-pooled
• Size: 3,999,008 training samples
• Columns: text and label
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	min: 32 tokens mean: 77.88 tokens max: 181 tokens	size: 768 elements

• Samples:

  text	label
  The book was then released as a movie of the same title. The 2014 Rose Parade will be a tribute to a world of dreams and the 101st Rose Bowl Game, which will take place at 1:30 p.m. PT / 4:30p.m. ET on New Year's Day Last minute tickets for the game can be purchased here. Highlights from the 2014 Rose Parade can be seen in the video below:	[1.3383212089538574, 1.189132809638977, -2.1790544986724854, -0.134178027510643, -1.6490687131881714, ...]
  Sign in to Azure. SQL Database provides relational data storage, access, and management services on Azure. To use it, you'll need an Azure subscription. Open a web browser, and browse to http://azure.microsoft.com/ . To get started with a free account, click free trial in the upper right corner and follow the steps.	[0.7683761119842529, 0.5112201571464539, -0.03011413663625717, -0.12728644907474518, 0.17182348668575287, ...]
  Kevin Costner celebrates turning 60. Happy birthday Kevin Costner! As the actor celebrates turning 60, HELLO! Online takes a look at what the stars have in store for Kevin and you…. This is what Kevin, who is a Capricorn, can be expecting today:	[0.26515713334083557, 2.5962588787078857, -1.8560686111450195, -0.6646427512168884, -1.7368828058242798, ...]

• Loss: EmbedDistillLoss with these parameters:

  {
    "distance_metric": "cosine",
    "projection_dim": null
  }
  

pubmedqa-query-gte-modernbert-pooled

pubmedqa-query-gte-modernbert-pooled

• Dataset: stephantulkens/pubmedqa-query-gte-modernbert-pooled
• Size: 272,518 training samples
• Columns: text and label
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	min: 10 tokens mean: 23.01 tokens max: 53 tokens	size: 768 elements

• Samples:

  text	label
  Lasting hand self-disinfection: A backup for hospital hand hygiene?	[0.007685241289436817, -0.43003925681114197, 1.1973797082901, -0.5023999214172363, 0.820604681968689, ...]
  Are stressful life events associated with a poor in-vitro fertilization ( IVF ) outcome : a prospective study?	[1.3466689586639404, -1.6600078344345093, -1.1551469564437866, -0.8789319396018982, 1.4987536668777466, ...]
  Does orbitofrontal dysfunction discriminate behavioral variant frontotemporal dementia from Alzheimer 's disease?	[0.05421882122755051, -0.059128329157829285, -1.1960982084274292, -1.6018415689468384, 0.589949369430542, ...]

• Loss: EmbedDistillLoss with these parameters:

  {
    "distance_metric": "cosine",
    "projection_dim": null
  }
  

gooaq-query-gte-modernbert-pooled

gooaq-query-gte-modernbert-pooled

• Dataset: stephantulkens/gooaq-query-gte-modernbert-pooled
• Size: 500,000 training samples
• Columns: text and label
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	min: 8 tokens mean: 12.29 tokens max: 22 tokens	size: 768 elements

• Samples:

  text	label
  what time does the dallas cowboys and washington redskins play today?	[-0.8482640981674194, 2.0467615127563477, -0.6735870838165283, -1.5679267644882202, 0.5012792348861694, ...]
  what is nwc in finance?	[0.4169943630695343, -0.4177152216434479, -0.8103370070457458, -0.18884222209453583, -2.4537534713745117, ...]
  what are silk road countries?	[-0.8498127460479736, -0.05223170295357704, -1.962998390197754, 0.139049232006073, -0.06982739269733429, ...]

• Loss: EmbedDistillLoss with these parameters:

  {
    "distance_metric": "cosine",
    "projection_dim": null
  }
  

squad-query-gte-modernbert-pooled

squad-query-gte-modernbert-pooled

• Dataset: stephantulkens/squad-query-gte-modernbert-pooled
• Size: 87,599 training samples
• Columns: text and label
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	min: 8 tokens mean: 14.71 tokens max: 33 tokens	size: 768 elements

• Samples:

  text	label
  What percentage of Egyptians polled support death penalty for those leaving Islam?	[-0.9493468999862671, -0.5618804693222046, 0.8554431200027466, -0.9016249179840088, 0.7195834517478943, ...]
  Ann Arbor ranks 1st among what goods sold?	[-0.0908542051911354, 0.6183239817619324, 0.6204090714454651, -1.5438891649246216, 1.8023393154144287, ...]
  In developing countries, who makes most of the spending decisions?	[0.16595937311649323, -0.7899607419967651, 0.13864848017692566, 0.473702073097229, 0.293387770652771, ...]

• Loss: EmbedDistillLoss with these parameters:

  {
    "distance_metric": "cosine",
    "projection_dim": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• world_size: 2 GPUs (256 training examples per step)
• num_train_epochs: 1
• learning_rate: 3e-05
• warmup_steps: 0.05
• bf16: True
• per_device_eval_batch_size: 32
• ddp_find_unused_parameters: False

Training used a two-phase setup. An optional warm-up first fitted the new 384-to-768 projection head on cached encoder outputs (warmhead.py), after which the full model was trained with EmbedDistillLoss. Texts longer than 256 tokenizer tokens were removed before training. The final run used the pre-fitted head (1,500 cosine-optimization steps at a learning rate of 1e-3) and therefore trained the full model for one pass over all 5,870,041 examples.

All Hyperparameters

Click to expand

• per_device_train_batch_size: 128
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 3e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Click to expand

Training Time

• Training: 7.4 hours
• Evaluation: 6.9 minutes
• Total: 7.5 hours

Framework Versions

• Python: 3.13.15
• Sentence Transformers: 6.1.0
• Transformers: 5.16.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.15.0
• Datasets: 5.1.0
• Tokenizers: 0.23.1

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

EmbedDistillLoss

@article{kim2023embeddistill,
    title={EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval},
    author={Kim, Seungyeon and Rawat, Ankit Singh and Zaheer, Manzil and Jayasumana, Sadeep and Sadhanala, Veeranjaneyulu and Jitkrittum, Wittawat and Menon, Aditya Krishna and Fergus, Rob and Kumar, Sanjiv},
    year={2023},
    eprint={2301.12005},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}