ModernBERT Embed base Legal Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("IoannisKat1/bge-m3-legal-matryoshka")
# Run inference
sentences = [
    'What are the consequences for unlawful interference with sensitive data?',
    "Failure to notify the Authority of file establishment or permit changes is punished by up to three years’ imprisonment and a fine of one to five million Drachmas.\nMaintaining a file without a permit or violating permit terms is punished by at least one year’s imprisonment and a fine of one to five million Drachmas.\nUnauthorized file interconnection or without permit is punished by up to three years’ imprisonment and a fine of one to five million Drachmas.\nUnlawful interference with personal data is punished by imprisonment and a fine; for sensitive data, at least one year’s imprisonment and a fine of one to ten million Drachmas.\nControllers who fail to comply with Authority decisions or violate data transfer rules face at least two years’ imprisonment and a fine of one to five million Drachmas.\nIf acts were committed for unlawful benefit or to cause harm, punishment is up to ten years’ imprisonment and a fine of two to ten million Drachmas.\nIf acts jeopardize democratic governance or national security, punishment is confinement in a penitentiary and a fine of five to ten million Drachmas.\nActs committed due to negligence result in at least three months’ imprisonment and a fine.\nIf the Controller is not a natural person, the responsible party is the representative or head of the organization with administrative or managerial duties.\nAuthorized members of the Authority may carry out preliminary investigations even without Prosecutor’s order for certain offenses.\nThe Authority's President must notify the Public Prosecutor of any offenses under investigation, forwarding all relevant evidence.\nPreliminary investigations must conclude within two months of charges, and trial must begin within three months of completion.\nContinuation of proceedings is allowed only once and for extremely important reasons, with adjournment not exceeding two months.\nFelonies under this law fall under the jurisdiction of the Court of Appeal.\n",
    'Any person who, in contravention of the provisions of this law or of the provisions of lawfully ratified multilateral international conventions on the protection of copyright, unlawfully makes a fixation of a work or of copies, reproduces them directly or indirectly, temporarily or permanently in any form, in whole or in part, translates, adapts, alters or transforms them, or distributes them to the public by sale or other means, or possesses with the intent of distributing them, rents, performs in public, broadcasts by radio or television or any other means, communicates to the public works or copies by any means, imports copies of a work illegally produced abroad without the consent of the author and, in general, exploits works, reproductions or copies being the object of copyright or acts against the moral right of the author to decide freely on the publication and the presentation of his work to the public without additions or deletions, shall be liable to imprisonment of no less than a year and to a fine from 2.900-15.000 Euro.\nWithout the permission of the performers: fixes their performance; directly or indirectly, temporarily or permanently reproduces by any means and form, in whole or in part, the fixation of their performance; distributes to the public the fixation of their performance or possesses them with the purpose of distribution; rents the fixation of their performance; broadcasts by radio and television by any means, the live performance, unless such broadcasting is rebroadcasting of a legitimate broadcasting; communicates to the public the live performance made by any means, except radio and television broadcasting; makes available to the public, by wire or wireless means, in such a way that members of the public may access them from a place and at a time individually chosen by them, the fixation of their performance.\nWithout the permission of phonogram producers (producers of sound recordings): directly or indirectly, temporarily or permanently reproduces by any means and form, in whole or in part, their phonograms; distributes to the public the above recordings, or possesses them with the purpose of distribution; rents the said recordings; makes available to the public, by wire or wireless means, in such a way that members of the public may access them from a place and at a time individually chosen by them, their phonograms; imports the said recordings produced abroad without their consent.\nWithout the permission of producers of audiovisual works (producers of visual or sound and visual recordings): directly or indirectly, temporarily or permanently reproduces by any means and form, in whole or in part, the original and the copies of their films; distributes to the public the above recordings, including the copies thereof, or possesses them with the purpose of distribution; rents the said recordings; makes available to the public, by wire or wireless means, in such a way that members of the public may access them from a place and at a time individually chosen by them, the original and the copies of their films; imports the said recordings produced abroad without their consent; broadcasts by radio or television by any means including satellite transmission and cable retransmission, as well as the communication to the public.\nWithout the permission of radio and television organizations: rebroadcasts their broadcasts by any means; presents their broadcasts to the public in places accessible to the public against payment of an entrance fee; fixes their broadcasts on sound or sound and visual recordings, regardless of whether the broadcasts are transmitted by wire or by the air, including by cable or satellite; directly or indirectly, temporarily or permanently reproduces by any means and form, in whole or in part, the fixation of their broadcasts; distributes to the public the recordings containing the fixation or their broadcasts; rents the recordings containing the fixation of their broadcasts; makes available to the public, by wire or wireless means, in such a way that members of the public may access them from a place and at a time individually chosen by them, the fixation of their broadcasts.\nIf the financial gain sought or the damage caused by the perpetration of an act listed in paragraphs (1) and (2), above, is particularly great, the sanction shall be not less than two years imprisonment and a fine of from 2 to 10 million drachmas. If the guilty party has perpetrated any of the aforementioned acts by profession or at a commercial scale or if the circumstances in connection with the perpetration of the act indicate that the guilty party poses a serious threat to the protection of copyright or related rights, the sanction shall be imprisonment of up to ten (10) years and a fine of from 5 to 10 million drachmas, together with the withdrawal of the trading license of the undertaking which has served as the vehicle for the act. The act shall be likewise deemed to have been perpetrated by way of standard practice if the guilty party has on a previous occasion been convicted of a contravention pursuant to the provisions of the Article or for a violation of the preceding copyright legislation and sentenced to a non-redeemable period of imprisonment. Any infringement of copyright and related rights in the form of felony is tried by the competent Three-member Court of Appeal for Felonies.\nAny person who did not pay the remuneration provided for by Article 18, paragraph (3) hereof to a collecting society is punished with the sanction of paragraph (1), (2) and (3). The same sentence is imposed on the debtor who, after the issuance of the decision of the one-member first instance court, does not submit the declaration under the provisions of article 18, par. 6, of this law.\nThe sanctions specified in paragraph (1), above, shall be applicable likewise to any person who: uses or distributes, or possesses with the intent to distribute, any system or means whose sole purpose is to facilitate the unpermitted removal or neutralization of a technical system used to protect a computer program; manufactures or imports or distributes, or possesses with intent to distribute, equipment and other materials utilizable for the reproduction of a work which do not conform to the specifications determined pursuant to Article 59 of this Law; manufactures or imports or distributes, or possesses with intent to distribute, objects which can thwart the efficacy of the above-mentioned specifications, or engages in an act which can have that result; reproduces or uses a work without utilizing the equipment or without applying the systems specified pursuant to Article 60 of this Law; distributes, or possesses with intent to distribute, a phonogram or film without the special mark or control label specified pursuant to Article 61 of this Law.\nWhere a sentence of imprisonment is imposed with the option of redeemability, the sum payable for the redemption shall be 10 times the sum specified as per the case in the Penal Code.\nWhere mitigating circumstances exist, the fine imposed shall not be less than half of the minimum fine imposable as per the case under this Law.\nAny person who proceeds to authorized temporary or permanent reproduction of the database, translation, adaptation, arrangement and any other alteration of the database, distribution to the public of the database or of copies thereof, communication, display or performance of the database to the public, is punished by imprisonment of at least one (1) year and a fine of one (1) to five (5) million drachmas.\nAny person who proceeds to extraction and/or re-utilization of the whole or of a substantial part of the contents of the database without the authorization of the author thereof, is punished by imprisonment of at least one (1) year and a fine of one (1) to five (5) million drachmas (article 12 of Directive 96/9).\nWhen the object of the infringement refers to computer software, the culpable character of the action, as described in paragraph 1 of article 65A and under the prerequisites provided there, is raised under the condition that the infringer proceeds in the unreserved payment of the administrative fee and the infringement concerns a quantity of up to 50 programs.\nWhen the object of infringement concerns recordings of sound in which a work protected by copyright law has been recorded, the unreserved payment of an administrative fee according to the stipulation of par.2 of article 65A and under the prerequisites provided there, the culpable character of the action is raised under the condition that the infringement concerns a quantity of up to five hundred (500) illegal sound recording carriers.\nThe payment of the administrative fee and the raising of the culpable character of the action, do not relieve the infringers from the duty of buying off the copyright and related rights or from the duty of compensating and paying the rest expenses to the holders of these rights, according to the provisions of the relevant laws.\nIn case of recidivism during the same financial year the administrative fee provided for by article 65A doubles.\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,580 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 17.16 tokens max: 39 tokens	min: 27 tokens mean: 714.22 tokens max: 2824 tokens

• Samples:

  anchor	positive
  At what time must the controller provide the data subject with the required information?	1.Where personal data relating to a data subject are collected from the data subject, the controller shall, at the time when personal data are obtained, provide the data subject with all of the following information: (a) the identity and the contact details of the controller and, where applicable, of the controller's representative; (b) the contact details of the data protection officer, where applicable; (c) the purposes of the processing for which the personal data are intended as well as the legal basis for the processing; 4.5.2016 L 119/40 (d) where the processing is based on point (f) of Article 6(1), the legitimate interests pursued by the controller or by a third party; (e) the recipients or categories of recipients of the personal data, if any; (f) where applicable, the fact that the controller intends to transfer personal data to a third country or international organisation and the existence or absence of an adequacy decision by the Commission, or in the case of trans...
  What information does the duty of professional secrecy cover during the term of office?	1.Each Member State shall provide by law for all of the following: (a) the establishment of each supervisory authority; 4.5.2016 L 119/66 (b) the qualifications and eligibility conditions required to be appointed as member of each supervisory authority; (c) the rules and procedures for the appointment of the member or members of each supervisory authority; (d) the duration of the term of the member or members of each supervisory authority of no less than four years, except for the first appointment after 24 May 2016, part of which may take place for a shorter period where that is necessary to protect the independence of the supervisory authority by means of a staggered appointment procedure; (e) whether and, if so, for how many terms the member or members of each supervisory authority is eligible for reappointment; (f) the conditions governing the obligations of the member or members and staff of each supervisory authority, prohibitions on actions, occupations and benefits inco...
  Under what circumstances should the controller be allowed to further process personal data?	The processing of personal data for purposes other than those for which the personal data were initially collected should be allowed only where the processing is compatible with the purposes for which the personal data were initially collected. In such a case, no legal basis separate from that which allowed the collection of the personal data is required. If the processing is necessary for the performance of a task carried out in the public interest or in the exercise of official authority vested in the controller, Union or Member State law may determine and specify the tasks and purposes for which the further processing should be regarded as compatible and lawful. Further processing for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes should be considered to be compatible lawful processing operations. The legal basis provided by Union or Member State law for the processing of personal data may also provide a legal basis for ...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• gradient_accumulation_steps: 2
• learning_rate: 2e-05
• num_train_epochs: 15
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 15
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.8.1
• Datasets: 4.0.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}