modernbert_embed_base Finetuned on Data

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What action was the plaintiff urged to complete within the next 24 hours?',
    '**Court (Civil/Criminal): Civil**\n\n**Provisions:**\n\n**Time of commission of the act:**\n\n**Outcome (not guilty, guilty):**\n\n**Rationale:**\n\n**Facts:**\nThe plaintiff holds credit card number ............ with the defendant banking corporation. Based on the application for alternative networks dated 19/7/2015 with number ......... submitted at a branch of the defendant, he was granted access to the electronic banking service (e-banking) to conduct banking transactions (debit, credit, updates, payments) remotely. On 30/11/2020, the plaintiff fell victim to electronic fraud through the "phishing" method, whereby an unknown perpetrator managed to withdraw a total amount of €3,121.75 from the aforementioned credit card. Specifically, the plaintiff received an email at 1:35 PM on 29/11/2020 from sender ...... with address ........, informing him that due to an impending system change, he needed to verify the mobile phone number linked to the credit card, urging him to complete the verification process within the next 24 hours by following a link titled ........; otherwise, his account would be locked for security reasons. The plaintiff read this email on the afternoon of 30 November 2020 and, believing it was from the defendant, followed the instructions and proceeded via the provided link to a website that was identical (a clone) to that of the defendant. On this page, he was asked to enter the six-digit security code (.........) that had just been sent to his mobile phone by the defendant at 3:41 PM, with the note that it was an activation code for his ........ card at ........., which he entered.\n\nSubsequently, the plaintiff received, according to his statements, a new email (not submitted), which requested him to enter the details of the aforementioned credit card, specifically the name of the cardholder and the card number, not the PIN, which he also entered, convinced that he was within the online environment of the defendant. Then, at 3:47 PM, he received a message on his mobile phone from the defendant containing the exact same content as the one he received at 3:41 PM, while at 3:50 PM he received a message stating that the activation of his ......... card at ....... had been completed. Once the plaintiff read this, he became concerned that something was not right, and immediately called (at 4:41 PM) the defendant\'s call center to inform them. There, the employees, with whom he finally connected at 5:04 PM due to high call center volume, advised him to delete the relevant emails, cancel his credit card, change his access passwords for the service, and submit a dispute request regarding the conducted transactions. The plaintiff electronically sent this request to the defendant, disputing the detailed transactions amounting to €3,121.75, which were conducted on 30/11/2020 during the time frame of 16:37:45-16:43:34 PM, arguing that he had neither performed them himself nor authorized anyone else to do so. The plaintiff specifically disputed the following transactions, as evidenced by the account activity of the disputed credit card during the aforementioned timeframe: a) transaction number ......... amounting to €150.62 conducted on 30/11/2020 at 4:43:34 PM, b) transaction number ........ amounting to €293.20 conducted on 30/11/2020 at 4:42:40 PM, c) transaction number ............ amounting to €295.21 conducted on 30/11/2020 at 4:42:10 PM, d) transaction number .......... amounting to €299.22 conducted on 30/11/2020 at 4:41:31 PM, e) transaction number ........ amounting to €297.21 conducted on 30/11/2020 at 4:41:01 PM, f) transaction number ........ amounting to €299.22 conducted on 30/11/2020 at 4:40:27 PM, g) transaction number ....... amounting to €299.22 conducted on 30/11/2020 at 4:39:55 PM, h) transaction number ...... amounting to €299.22 conducted on 30/11/2020 at 4:39:22 PM, i) transaction number ......... amounting to €297.22 conducted on 30/11/2020 at 4:38:52 PM, j) transaction number ......... amounting to €295.21 conducted on 30/11/2020 at 4:38:17 PM, and k) transaction number ......... amounting to €296.21 conducted on 30/11/2020 at 4:37:45 PM. In its response letter dated 21/12/2020, the defendant denied responsibility for the costs of the aforementioned transactions, placing the entire blame on the plaintiff for the leak of his card details and security code to the fraudulent page. The plaintiff, completely denying any fault for the conducted transactions, repeatedly contacted the defendant, both by phone and via email (see emails dated 15/1/2021 and 11/2/2021), while on 2/3/2021, he electronically sent a report dated 1/03/2021 to the Consumer Advocate’s email address, recounting the events and requesting that the aforementioned Independent Authority intervene to have the disputed debt canceled. In its letter with reference number ...../27.04.2021, the aforementioned Independent Authority informed the plaintiff that the case was outside its mediating role and was therefore archived. Subsequently, the plaintiff sent the defendant on 5/3/2021 his extrajudicial statement dated 4/3/2021, calling upon it to fully cancel the debt of €3,121.75 that had been unjustly incurred against him within two days and to immediately instruct the representatives of the collection agency working with it to cease contacting him regarding the disputed case. The defendant sent the plaintiff a message on his mobile phone on 20/04/2021 informing him that his case was still being processed due to lengthy operational requirements, while on 23/04/2021, via email, it informed him that considering their good cooperation and his efforts to keep them updated, it had reviewed his case and decided to refund him the amounts of the transactions that were conducted after his contact with their representatives on 30/11/2020 at 4:41 PM, totaling €1,038.25, specifically the following: a) transaction of €150.62 conducted on 30/11/2020 at 4:43 PM, b) transaction of €295.21 conducted on 30/11/2020 at 4:42 PM, c) transaction of €293.20 conducted on 30/11/2020 at 4:42 PM, and d) transaction of €299.22 conducted on 30/11/2020 at 4:41 PM. Beyond this, the defendant refused to refund the plaintiff the amount of the remaining transactions conducted on 30/11/2020, totaling €2,376.08 (and not €2,376.48 as incorrectly stated by the plaintiff in his lawsuit), which the plaintiff ultimately fully paid, transferring €2,342.77 to the defendant on 7/06/2021 and €33.31 on 15/06/2021 (see related deposit receipts).',
    "1.Without prejudice to other tasks set out under this Regulation, each supervisory authority shall on its territory: (a)  monitor and enforce the application of this Regulation; (b)  promote public awareness and understanding of the risks, rules, safeguards and rights in relation to processing. Activities addressed specifically to children shall receive specific attention; (c)  advise, in accordance with Member State law, the national parliament, the government, and other institutions and bodies on legislative and administrative measures relating to the protection of natural persons' rights and freedoms with regard to processing; (d)  promote the awareness of controllers and processors of their obligations under this Regulation; (e)  upon request, provide information to any data subject concerning the exercise of their rights under this Regulation and, if appropriate, cooperate with the supervisory authorities in other Member States to that end; (f)  handle complaints lodged by a data subject, or by a body, organisation or association in accordance with Article 80, and investigate, to the extent appropriate, the subject matter of the complaint and inform the complainant of the progress and the outcome of the investigation within a reasonable period, in particular if further investigation or coordination with another supervisory authority is necessary; (g)  cooperate with, including sharing information and provide mutual assistance to, other supervisory authorities with a view to ensuring the consistency of application and enforcement of this Regulation; (h)  conduct investigations on the application of this Regulation, including on the basis of information received from another supervisory authority or other public authority; (i)  monitor relevant developments, insofar as they have an impact on the protection of personal data, in particular the development of information and communication technologies and commercial practices; (j)  adopt standard contractual clauses referred to in Article 28(8) and in point (d) of Article 46(2); (k)  establish and maintain a list in relation to the requirement for data protection impact assessment pursuant to Article 35(4); (l)  give advice on the processing operations referred to in Article 36(2); (m)  encourage the drawing up of codes of conduct pursuant to Article 40(1) and provide an opinion and approve such codes of conduct which provide sufficient safeguards, pursuant to Article 40(5); (n)  encourage the establishment of data protection certification mechanisms and of data protection seals and marks pursuant to Article 42(1), and approve the criteria of certification pursuant to Article 42(5); (o)  where applicable, carry out a periodic review of certifications issued in accordance with Article 42(7); 4.5.2016 L 119/68   (p)  draft and publish the criteria for accreditation of a body for monitoring codes of conduct pursuant to Article 41 and of a certification body pursuant to Article 43; (q)  conduct the accreditation of a body for monitoring codes of conduct pursuant to Article 41 and of a certification body pursuant to Article 43; (r)  authorise contractual clauses and provisions referred to in Article 46(3); (s)  approve binding corporate rules pursuant to Article 47; (t)  contribute to the activities of the Board; (u)  keep internal records of infringements of this Regulation and of measures taken in accordance with Article 58(2); and (v)  fulfil any other tasks related to the protection of personal data.\n2.Each supervisory authority shall facilitate the submission of complaints referred to in point (f) of paragraph 1 by measures such as a complaint submission form which can also be completed electronically, without excluding other means of communication.\n3.The performance of the tasks of each supervisory authority shall be free of charge for the data subject and, where applicable, for the data protection officer.\n4.Where requests are manifestly unfounded or excessive, in particular because of their repetitive character, the supervisory authority may charge a reasonable fee based on administrative costs, or refuse to act on the request. The supervisory authority shall bear the burden of demonstrating the manifestly unfounded or excessive character of the request.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.3932, -0.0156],
#         [ 0.3932,  1.0000, -0.0606],
#         [-0.0156, -0.0606,  1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,627 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 15.44 tokens max: 35 tokens	min: 25 tokens mean: 627.36 tokens max: 2429 tokens

• Samples:

  anchor	positive
  What type of data processing triggers the obligations for entities with fewer than 250 employees?	1.Each controller and, where applicable, the controller's representative, shall maintain a record of processing activities under its responsibility. That record shall contain all of the following information: (a) the name and contact details of the controller and, where applicable, the joint controller, the controller's representative and the data protection officer; (b) the purposes of the processing; (c) a description of the categories of data subjects and of the categories of personal data; 4.5.2016 L 119/50 (d) the categories of recipients to whom the personal data have been or will be disclosed including recipients in third countries or international organisations; (e) where applicable, transfers of personal data to a third country or an international organisation, including the identification of that third country or international organisation and, in the case of transfers referred to in the second subparagraph of Article 49(1), the documentation of suitable safeguards; (f) ...
  For what purposes can the data subject object to data processing based on personal grounds?	1.The data subject shall have the right to object, on grounds relating to his or her particular situation, at any time to processing of personal data concerning him or her which is based on point (e) or (f) of Article 6(1), including profiling based on those provisions. The controller shall no longer process the personal data unless the controller demonstrates compelling legitimate grounds for the processing which override the interests, rights and freedoms of the data subject or for the establishment, exercise or defence of legal claims. 2.Where personal data are processed for direct marketing purposes, the data subject shall have the right to object at any time to processing of personal data concerning him or her for such marketing, which includes profiling to the extent that it is related to such direct marketing. 3.Where the data subject objects to processing for direct marketing purposes, the personal data shall no longer be processed for such purposes. 4.5.2016 L 119/45 4.At th...
  What should legally binding measures refer to?	In order to ensure consistent monitoring and enforcement of this Regulation throughout the Union, the supervisory authorities should have in each Member State the same tasks and effective powers, including powers of investigation, corrective powers and sanctions, and authorisation and advisory powers, in particular in cases of complaints from natural persons, and without prejudice to the powers of prosecutorial authorities under Member State law, to bring infringements of this Regulation to the attention of the judicial authorities and engage in legal proceedings. Such powers should also include the power to impose a temporary or definitive limitation, including a ban, on processing. Member States may specify other tasks related to the protection of personal data under this Regulation. The powers of supervisory authorities should be exercised in accordance with appropriate procedural safeguards set out in Union and Member State law, impartially, fairly and within a reasonable time. In ...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• gradient_accumulation_steps: 2
• learning_rate: 2e-05
• num_train_epochs: 10
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.1.2
• Transformers: 4.51.3
• PyTorch: 2.8.0+cu126
• Accelerate: 1.11.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}