BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("cristiano-sartori/bge_ft")
# Run inference
sentences = [
    'An expression is referentially transparent if it always returns the same value, no matter\nthe global state of the program. A referentially transparent expression can be replaced by its value without\nchanging the result of the program.\nSay we have a value representing a class of students and their GPAs. Given the following defintions:\n1 case class Student(gpa: Double)\n2\n3 def count(c: List[Student], student: Student): Double =\n4 c.filter(s => s == student).size\n5\n6 val students = List(\n7 Student(1.0), Student(2.0), Student(3.0),\n8 Student(4.0), Student(5.0), Student(6.0)\n9 )\nAnd the expression e:\n1 count(students, Student(6.0))',
    "Let's break this down simply. The function `count` takes a list of students and a specific student, then counts how many times that student appears in the list. In our example, we have a list of students with different GPAs.\n\nWhen we call `count(students, Student(6.0))`, we are asking how many times a student with a GPA of 6.0 is in our list. Since we have `Student(6.0)` in the list only once, the function will return 1.\n\nNow, to understand referential transparency: if we replace the call `count(students, Student(6.0))` with its value (which is 1), the overall result of the program would still remain the same. So, the expression is referentially transparent because it consistently gives us the same output (1) regardless of the program's state.",
    'To solve the problem of identifying a non-empty subset \\( S \\subsetneq V \\) in a \\( d \\)-regular graph \\( G \\) using the second eigenvector \\( v_2 \\) of the normalized adjacency matrix \\( M \\), we can follow these steps:\n\n### Step 1: Understanding Eigenvector \\( v_2 \\)\n\nThe second eigenvector \\( v_2 \\) is orthogonal to the all-ones vector \\( v_1 \\), indicating that it captures structural features of the graph related to its connected components. Its entries will have both positive and negative values, allowing us to partition the vertices.\n\n### Step 2: Properties of \\( v_2 \\)\n\n- The orthogonality to \\( v_1 \\) ensures that there are vertices with positive values (indicating one group) and negative values (indicating another group). Therefore, we can define two sets based on the sign of the entries in \\( v_2 \\).\n\n### Step 3: Designing the Procedure\n\n1. **Define the Sets:**\n   - Let \\( S = \\{ i \\in V : v_2(i) > 0 \\} \\).\n   - Let \\( T = \\{ i \\in V : v_2(i) < 0 \\} \\).\n\n2. **Check for Non-emptiness:**\n   - Since \\( v_2 \\) is orthogonal to \\( v_1 \\), at least one vertex must have a positive value and at least one must have a negative value. Hence, \\( S \\) cannot be empty, and \\( S \\neq V \\).\n\n### Step 4: Showing that \\( S \\) Cuts 0 Edges\n\nWe need to demonstrate that the cut defined by \\( S \\) does not cross any edges:\n\n- **Edge Contributions:**\n  - For any edge \\( (i, j) \\) in the graph, if one vertex belongs to \\( S \\) and the other to \\( T \\), the eigenvalue relationship \\( M \\cdot v_2 = v_2 \\) indicates that the edge would create a mismatched contribution, leading to a contradiction. This implies that no edges can exist between \\( S \\) and \\( T \\).\n\n### Final Procedure\n\nThe procedure can be summarized as follows:\n\n```plaintext\nProcedure FindDisconnectedSet(v_2):\n    S = { i ∈ V : v_2(i) > 0 }\n    T = { i ∈ V : v_2(i) < 0 }\n    \n    if S is empty:\n        return T\n    else:\n        return S\n```\n\n### Conclusion\n\nThis algorithm ensures that we find a non-empty subset \\( S \\subsetneq V \\) that defines a cut with no edges crossing between \\( S \\) and \\( V \\setminus S \\), under the condition that \\( \\lambda_2 = 1 \\).',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,140 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 5 tokens mean: 168.34 tokens max: 512 tokens	min: 25 tokens mean: 374.7 tokens max: 512 tokens

• Samples:

  anchor	positive
  Consider the task of classifying reviews as positive or negative. To create a reference for this task, two human annotators were asked to rate 1000 movie reviews as positive or negative.The first annotator rated {a} reviews as positive and the rest as negative. The second annotator rated {b} reviews as positive and the rest as negative. 80 reviews were rated as positive by both annotators. What is the raw agreement between the two annotators?Give your answer as a numerical value to three decimal places.	To calculate the raw agreement between the two annotators, we can use the following formula: [ \text{Raw Agreement} = \frac{\text{Number of agreements}}{\text{Total number of reviews}} ] 1. Identify the total number of reviews: In this case, it is given that there are 1000 movie reviews. 2. Identify the number of agreements: The agreements consist of the reviews that both annotators rated as positive or both rated as negative. We know that: - Both annotators rated 80 reviews as positive. - To find the number of reviews both rated as negative, we need to calculate how many reviews each annotator rated as negative. Let’s denote: - ( a ): the number of positive reviews by Annotator 1 - ( b ): the number of positive reviews by Annotator 2 Thus, the number of negative reviews for each annotator would be: - Negative reviews by Annotator 1 = ( 1000 - a ) - Negative reviews by Annotator 2 = ( 1000 - b ) 3. Calculate the total agreements: -...
  Let $y_1, y_2, \ldots, y_n$ be uniform random bits. For each non-empty subset $S\subseteq {1,2, \ldots, n}$, define $X_S = \oplus_{i\in S}:y_i$. Show that the bits ${X_S: \emptyset \neq S\subseteq {1,2, \ldots, n} }$ are pairwise independent. This shows how to stretch $n$ truly random bits to $2^n-1$ pairwise independent bits. \ \emph{Hint: Observe that it is sufficient to prove $\mathbb{E}[X_S] = 1/2$ and $\mathbb{E}[X_S X_T] = 1/4$ to show that they are pairwise independent. Also use the identity $\oplus_{i\in A}: y_i = \frac{1}{2}\left( 1 - \prod_{i\in A} (-1)^{y_i} \right)$.}	To demonstrate that the random variables ( {X_S : S \subseteq {1, 2, \ldots, n}, S \neq \emptyset} ) are pairwise independent, we need to show two things: 1. The expected value ( \mathbb{E}[X_S] = \frac{1}{2} ) for any non-empty subset ( S ). 2. The expected value of the product of any two variables ( X_S ) and ( X_T ) (where ( S ) and ( T ) are non-empty subsets of ({1,2,\ldots,n})) satisfies ( \mathbb{E}[X_S X_T] = \frac{1}{4} ). ### Step 1: Calculate ( \mathbb{E}[X_S] ) The random variable ( X_S ) is defined as the XOR (exclusive OR) of bits indexed by elements of ( S ): [ X_S = \oplus_{i \in S} y_i ] For each ( y_i ), since it is a uniform random bit, we have: [ \mathbb{E}[y_i] = \frac{1}{2} ] The XOR operation ( X_S = y_{i_1} \oplus y_{i_2} \oplus \ldots \oplus y_{i_k} ) (where ( S = {i_1, i_2, \ldots, i_k} )) can take the value 0 or 1. The expected value of ( X_S ) can be computed as follows: 1. The outcome ( X_S = 0 ) occurs ...
  We have a collection of rectangles in a plane, whose sides are aligned with the coordinate axes. Each rectangle is represented by its lower left corner $(x_1,y_1)$ and its upper right corner $(x_2,y_2)$. All coordinates are of type Long. We require $x_1 \le x_2$ and $y_1 \le y_2$. Define a case class Rectangle storing two corners.	### Summary To represent rectangles in a plane with aligned sides, we can define a case class in Scala that captures the necessary properties while enforcing the required constraints on the coordinates. Each rectangle will be defined by its lower left corner ((x_1, y_1)) and its upper right corner ((x_2, y_2)). We will ensure that (x_1 \le x_2) and (y_1 \le y_2) through constructor validation. ### Implementation Here’s a concise implementation of the Rectangle case class with validation: scala<br>case class Rectangle(x1: Long, y1: Long, x2: Long, y2: Long) {<br> require(x1 <= x2, "x1 must be less than or equal to x2")<br> require(y1 <= y2, "y1 must be less than or equal to y2")<br>}<br> ### Explanation 1. Case Class Definition: The Rectangle case class is defined with four parameters: x1, y1, x2, and y2, all of type Long. 2. Constraints Enforcement: The require statements in the constructor ensure that the specified conditions (x_1 \le x_2) and (y_1 ...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.8
• Sentence Transformers: 4.1.0
• Transformers: 4.52.4
• PyTorch: 2.7.0+cu126
• Accelerate: 1.3.0
• Datasets: 3.6.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}