E5-Mistral-7B-Instruct QLoRA Financial Retrieval

This is a sentence-transformers model finetuned from intfloat/e5-mistral-7b-instruct on the fin_modern_bert-pairs-sec-synthetic-v1 dataset. It maps inputs to a 4096-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/e5-mistral-7b-instruct
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 4096 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • fin_modern_bert-pairs-sec-synthetic-v1
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'MistralModel'})
  (1): Pooling({'embedding_dimension': 4096, 'pooling_mode': 'lasttoken', 'include_prompt': True})
  (2): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kokitabayashi/e5-mistral-qlora_FinModernBERT_query")
# Run inference
queries = [
    'Where can I find the computation for the Ratio of Earnings to Fixed Charges?',
]
documents = [
    '12.1 Computation of Ratio of Earnings to Fixed Charges.',
    'Total Utility Operating Revenues $ 3,011,574',
    'Net income per common share -- diluted $ 1.60 $ 1.64 $ 1.48 $ 1.30',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 4096] [3, 4096]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.9219, 0.0568, 0.0930]])

Evaluation

Metrics

Information Retrieval

• Dataset: finance-eval
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "query_prompt": "Instruct: Given a financial question, retrieve relevant financial passages that answer the question\nQuery: "
  }
  

Training Details

Training Dataset

fin_modern_bert-pairs-sec-synthetic-v1

• Dataset: fin_modern_bert-pairs-sec-synthetic-v1
• Size: 44,709 training samples
• Columns: query and positive
• Approximate statistics based on the first 100 samples:

  	query	positive
  type	string	string
  modality	text	text
  details	min: 16 tokens mean: 31.38 tokens max: 58 tokens	min: 17 tokens mean: 74.24 tokens max: 222 tokens

• Samples:

  query	positive
  What was the net income for Bank of America Corporation for the fiscal year 2012?	Net income $ 4,188 $ 4,188
  What specific changes were made to the Company's credit agreement in April 2021 regarding its revolving credit facility?	The credit agreement amends the Company’s existing revolving credit facility to add a new five-year, delayed draw, term loan for up to $800 million.
  What are the SEC file numbers for the Ameren Companies' filings?	The file number references for the Ameren Companies’ filings with the SEC are: Ameren, 1-14756; Ameren Missouri, 1-2967; Ameren Illinois, 1-3672; and Genco, 333-56594.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 2,
      "mini_batch_num_tokens": null,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

fin_modern_bert-pairs-sec-synthetic-v1

• Dataset: fin_modern_bert-pairs-sec-synthetic-v1
• Size: 9,580 evaluation samples
• Columns: query and positive
• Approximate statistics based on the first 100 samples:

  	query	positive
  type	string	string
  modality	text	text
  details	min: 12 tokens mean: 32.1 tokens max: 58 tokens	min: 18 tokens mean: 78.29 tokens max: 329 tokens

• Samples:

  query	positive
  What were the specific terms and maturity dates of the unsecured notes issued by the Company in September 2012 and December 2012?	In September 2012, the Company issued $450,000,000 principal amount of unsecured notes in a public offering under its existing shelf registration statement. The notes mature in September 2022 and were issued at a 2.95% rate. In December 2012, the Company issued $250,000,000 principal amount of unsecured notes in a public offering under its existing shelf registration statement. The notes mature in March 2023 and were issued at a 2.85% rate.
  What was the estimated fair value of the company's debt as of December 31, 2005, and how does it compare to the carrying amount?	As of December 31, 2005 and 2004, the carrying amount of debt, net of capital leases, was $10,516 and $11,884 and the fair value of debt, based on current market rates for debt of the same risk and maturities, was estimated at $11,643 and $13,198.
  What was the total amount of assets managed by the Asset Management segment for owned assets as of December 31, 2016?	At December 31, 2016, the Asset Management segment managed $36.2 billion of owned assets.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 2,
      "mini_batch_num_tokens": null,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• num_train_epochs: 4
• warmup_steps: 0.1
• bf16: True
• logging_nan_inf_filter: False
• per_device_eval_batch_size: 32
• prompts: {'query': 'Instruct: Given a financial question, retrieve relevant financial passages that answer the question\nQuery: ', 'positive': ''}
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 32
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: False
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: {'query': 'Instruct: Given a financial question, retrieve relevant financial passages that answer the question\nQuery: ', 'positive': ''}
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Click to expand

Training Time

• Training: 6.5 hours

Framework Versions

• Python: 3.13.16
• Sentence Transformers: 6.2.0
• Transformers: 5.22.1
• PyTorch: 2.14.1+cu130
• Accelerate: 1.15.0
• Datasets: 4.8.5
• Tokenizers: 0.23.3

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}