e5-mistral-7b-instruct base trained on BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1

This is a sentence-transformers model finetuned from intfloat/e5-mistral-7b-instruct on the fin_modern_bert-pairs-sec-synthetic-v1 dataset. It maps sentences & paragraphs to a 4096-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/e5-mistral-7b-instruct
• Maximum Sequence Length: 32768 tokens
• Output Dimensionality: 4096 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • fin_modern_bert-pairs-sec-synthetic-v1
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'MistralModel'})
  (1): Pooling({'embedding_dimension': 4096, 'pooling_mode': 'lasttoken', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kokitabayashi/e5-mistral_FinModernBERT_query")
# Run inference
queries = [
    'What were the total capital expenditures for capacity expansion projects and the amount of cash returned to shareholders in fiscal 2015?',
]
documents = [
    'In fiscal 2015, our return on average invested capital improved modestly, despite near record levels of capital spending of $120 million in our capacity expansion projects... Also during fiscal 2015, we returned $718 million in cash to our shareholders through dividend payments of $256 million and share repurchases of $462 million, while maintaining investment-grade credit ratings.',
    'BNO will be subject to credit risk with respect to counterparties to OTC contracts entered into by BNO or held by special purpose or structured vehicles. Unlike in futures contracts, the counterparty to these contracts is generally a single bank or other financial institution, rather than a clearing organization backed by a group of financial institutions. As a result, there will be greater counterparty credit risk in these transactions.',
    'The decrease in core unit volume in 2005 reflects the following factors: \x95 Loss of market share in our North American roll materials business following our implementation of selling price increases to offset higher raw material costs \x95 The impact of an extra week in the 2004 fiscal year \x95 Accelerated purchases by Office and Consumer Products customers in advance of our 2005 selling price increases, which contributed to higher growth in 2004',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 4096] [3, 4096]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[1.0000, 1.0000, 0.9999]])

Evaluation

Metrics

Information Retrieval

• Dataset: finance-eval
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

fin_modern_bert-pairs-sec-synthetic-v1

• Dataset: fin_modern_bert-pairs-sec-synthetic-v1
• Size: 44,709 training samples
• Columns: query and positive
• Approximate statistics based on the first 100 samples:

  	query	positive
  type	string	string
  modality	text	text
  details	min: 15 tokens mean: 30.88 tokens max: 50 tokens	min: 14 tokens mean: 79.35 tokens max: 277 tokens

• Samples:

  query	positive
  What were the primary reasons for the decline in gross margins in 2014 compared to 2013?	Gross margins declined 1 percentage point in 2014 compared to 2013 on slightly higher revenue. Higher inventory charges and an increase in three year warranty costs were the primary reasons for the lower gross margins.
  What is the specific mechanism BNO uses to ensure that losses do not accumulate in its futures accounts?	Futures Contracts are marked to market at the end of each trading day and the margin required with respect to such contracts is adjusted accordingly. This process of marking-to-market is designed to prevent losses from accumulating in any futures account.
  What were the total net charges recognized for the PTI (Process to Improve) initiative in 2009, and what were the primary drivers of the accelerated depreciation and asset impairment costs?	Net charges for the PTI were $31 million in 2009. Most of the accelerated depreciation, asset impairment charges and other shutdown costs primarily relate to the rationalization of the manufacturing network in the BioPharmaceuticals segment.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

fin_modern_bert-pairs-sec-synthetic-v1

• Dataset: fin_modern_bert-pairs-sec-synthetic-v1
• Size: 9,580 evaluation samples
• Columns: query and positive
• Approximate statistics based on the first 100 samples:

  	query	positive
  type	string	string
  modality	text	text
  details	min: 18 tokens mean: 31.56 tokens max: 48 tokens	min: 14 tokens mean: 70.35 tokens max: 301 tokens

• Samples:

  query	positive
  What specific factors contributed to the $88 million increase in Ameren Missouri's electric margins in 2012 compared to 2011?	Ameren Missouri's electric margins increased by $88 million, or 4%, in 2012 compared with 2011. The following items had a favorable impact on Ameren Missouri's electric margins: Higher electric base rates, effective July 2011 as a result of the 2011 MoPSC electric rate order, which increased revenues by $102 million, partially offset by an increase in net energy costs of $25 million. Reduced purchased power expense as a result of a FERC-ordered refund received from Entergy in 2012 relating to a power purchase agreement that expired in 2009, which increased margins by $24 million. The absence in 2012 of a reduction in revenues recorded in 2011 resulting from the MoPSC's April 2011 FAC prudence review order. The recovery of labor and benefit costs for crews assisting with Hurricane Sandy power restoration, which increased revenues by $7 million and was fully offset by a related increase in operations and maintenance costs, with no overall impact on net income. Higher transmission service...
  What was the net cash provided from operating activities for the fiscal year ended December 31, 2009?	Net cash provided from operating activities: 358,525 (in thousands)
  What are the maturity dates and interest rates for the notes listed in the filing?	4n. Form of 5.875% Notes due 2036... 4o. Form of 4.375% Notes due 2016... 4p. Form of 4.625% Notes due 2021... 4q. Form of 5.45% Notes due 2018... 4r. Form of 6.125% Notes due 2038.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 2
• num_train_epochs: 1
• warmup_steps: 0.1
• gradient_accumulation_steps: 32
• bf16: True
• per_device_eval_batch_size: 2
• dataloader_num_workers: 2
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 2
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 32
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 2
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Training Time

• Training: 3.2 hours

Framework Versions

• Python: 3.13.16
• Sentence Transformers: 5.7.0
• Transformers: 5.22.0
• PyTorch: 2.11.0+cu130
• Accelerate: 1.15.0
• Datasets: 4.8.5
• Tokenizers: 0.23.3

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}