BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from NovaSearch/stella_en_400M_v5 on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: NovaSearch/stella_en_400M_v5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: NewModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 1024, 'out_features': 1024, 'bias': True, 'activation_function': 'torch.nn.modules.linear.Identity'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("cristiano-sartori/stella_finetuned2")
# Run inference
sentences = [
    "In this problem we are going to investigate the linear programming relaxation of a classical scheduling problem. In the considered problem, we are given a set $M$ of $m$ machines and a set $J$ of $n$ jobs. Each job $j\\in J$ has a processing time $p_j > 0$ and can be processed on a subset $N(j) \\subseteq M$ of the machines.  The goal is to assign each job $j$ to a machine in $N(j)$ so as to complete all the jobs by a given deadline $T$. (Each machine can only process one job at a time.) If we, for $j\\in J$ and $i\\in N(j)$, let $x_{ij}$ denote the indicator variable indicating that $j$ was assigned to $i$, then we can formulate the scheduling problem as the following integer linear program: \\begin{align*} \\sum_{i\\in N(j)} x_{ij} & = 1 \\qquad \\mbox{for all }  j\\in J & \\hspace{-3em} \\mbox{\\small \\emph{(Each job $j$ should be assigned to a machine $i\\in N(j)$)}} \\\\ \\sum_{j\\in J: i \\in N(j)} x_{ij} p_j & \\leq T \\qquad \\mbox{for all } i \\in M & \\hspace{-3em} \\mbox{\\small \\emph{(Time needed to process jobs assigned to $i$ should be $\\leq T$)}} \\\\ x_{ij} &\\in \\{0,1\\}   \\  \\mbox{for all } j\\in J, \\ i \\in N(j) \\end{align*} The above integer linear program  is NP-hard to solve, but we can obtain a linear programming relaxation by relaxing the constraints $x_{ij} \\in \\{0,1\\}$ to $x_{ij} \\in [0,1]$. The obtained linear program can be solved in polynomial time using e.g. the ellipsoid method. \\\\[2mm] \\emph{Example.} An example is as follows. We have two machines $M = \\{m_1, m_2\\}$ and three jobs $J= \\{j_1, j_2, j_3\\}$. Job $j_1$ has processing time $1/2$ and can only be assigned to $m_1$;  job $j_2$ has processing time $1/2$ and can only be assigned to $m_2$; and job $j_3$ has processing time $1$ and can be assigned to either machine. Finally, we have the ``deadline'' $T=1$. An extreme point solution to the linear programming relaxation is $x^*_{11} = 1, x^*_{22} =1, x^*_{13} = 1/2$ and $x^*_{23} = 1/2$. The associated graph $H$ (defined in subproblem~\\textbf{a}) can be illustrated as follows: \\begin{tikzpicture} \\node[vertex] (a1) at (0,1.7) {$a_1$}; \\node[vertex] (a2) at (0,0.3) {$a_2$}; \\node[vertex] (b1) at (3,2.5) {$b_1$}; \\node[vertex] (b2) at (3,1) {$b_2$}; \\node[vertex] (b3) at (3,-0.5) {$b_3$}; \\draw (a1) edge (b3); \\draw (a2) edge (b3); \\end{tikzpicture} Use the structural result proved in the first subproblem to devise an efficient rounding algorithm that, given an instance and a feasible extreme point $x^*$ in the linear programming relaxation corresponding to the instance, returns a schedule that completes all jobs by deadline $T + \\max_{j\\in J} p_j$. In other words, you wish to assign jobs to machines so that the total processing time of the jobs a machine receives is at most $T + \\max_{j\\in J} p_j$.",
    "To devise an efficient rounding algorithm based on the linear programming relaxation for the scheduling problem described, we need to follow these steps:\n\n### Step 1: Understand the Linear Programming Relaxation\n\nGiven the linear programming relaxation of the scheduling problem, we have:\n- Variables \\(x_{ij}\\) representing the fraction of job \\(j\\) assigned to machine \\(i\\).\n- Constraints ensuring each job is assigned exactly once and that the total processing time assigned to each machine does not exceed the deadline \\(T\\).\n\n### Step 2: Identify the Maximum Processing Time\n\nCalculate \\(P = \\max_{j \\in J} p_j\\), which is the maximum processing time of all jobs. This will be used to adjust the deadline in the rounding algorithm.\n\n### Step 3: Define the New Deadline\n\nThe new effective deadline for scheduling jobs becomes \\(T' = T + P\\).\n\n### Step 4: Create a Rounding Scheme\n\n1. **Initialization**: Start by initializing an empty schedule for each machine.\n2. **Job Assignment**:\n   - For each job \\(j\\):\n     - If \\(x_{ij} > 0\\) for a machine \\(i\\), assign job \\(j\\) to machine \\(i\\) with a specific probability based on the value of \\(x_{ij}\\):\n       - Assign job \\(j\\) to machine \\(i\\) with probability \\(x_{ij}\\).\n       - Use a random number generator to determine if the job is assigned based on the value of \\(x_{ij}\\).\n       - If \\(x_{ij} = 1\\), assign job \\(j\\) to machine \\(i\\) deterministically.\n3. **Check Capacity**: After assigning jobs to machines, check if the total processing time assigned to each machine exceeds \\(T'\\):\n   - If the total exceeds \\(T'\\), adjust the assignments. This can be done by removing the job with the largest processing time until the total processing time is within limits.\n4. **Output the Schedule**: The output schedule contains jobs assigned to each machine, ensuring that the total processing time does not exceed \\(T'\\).\n\n### Step 5: Ensure Feasibility of the Schedule\n\nThe rounding algorithm is designed to ensure that:\n- Each job is assigned to a machine in its feasible set \\(N(j)\\).\n- The total processing time for each machine does not exceed \\(T + P\\).\n\n### Example Application\n\nLet’s illustrate the rounding algorithm with the provided example:\n\n- Machines: \\(M = \\{m_1, m_2\\}\\)\n- Jobs: \\(J = \\{j_1, j_2, j_3\\}\\)\n- Processing times: \\(p_1 = 1/2\\), \\(p_2 = 1/2\\), \\(p_3 = 1\\)\n- Assignments: \\(x^*_{11} = 1\\), \\(x^*_{22} = 1\\), \\(x^*_{13} = 1/2\\), \\(x^*_{23} = 1/2\\)\n- Deadline: \\(T = 1\\)\n- Maximum processing time: \\(P = 1\\)\n- New deadline: \\(T' = 2\\)\n\n**Rounding Steps**:\n1. Assign \\(j_1\\) to \\(m_1\\) (since \\(x_{11} = 1\\)).\n2. Assign \\(j_2\\) to \\(m_2\\) (since \\(x_{22} = 1\\)).\n3. For job \\(j_3\\):\n   - Assign with probability \\(1/2\\) to either machine. Let’s say it’s assigned to \\(m_1\\).\n4. Check machine \\(m_1\\):\n   - Total processing time = \\(p_1 + p_3 = 1/2 + 1 = 1.5 \\leq 2\\) (valid).\n5. Check machine \\(m_2\\):\n   - Total processing time = \\(p_2 = 1/2 \\leq 2\\) (valid).\n\n### Conclusion\n\nThe proposed rounding algorithm yields a feasible schedule that respects the new deadline \\(T'\\) while ensuring that jobs are assigned appropriately to machines. This approach leverages the structural properties of the linear programming solution, providing a practical method for achieving a valid job schedule.",
    "It's best to use branching strategies instead of maintaining separate repositories for old versions. Create a branch for each version in the main repository, allowing you to easily apply bug fixes and features across versions. This way, you can maintain a clean history, facilitate code reviews, and simplify merging changes. Additionally, using tags to mark releases can help track versions effectively while keeping the main repository organized. This approach enhances collaboration and minimizes the risk of discrepancies between versions.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,140 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 5 tokens mean: 175.46 tokens max: 512 tokens	min: 13 tokens mean: 375.67 tokens max: 512 tokens

• Samples:

  anchor	positive
  Review the notion of depth seen in the lecture. What does it represent? Below is a formula for the depth of a divide and conquer algorithm working on an array segment of size $L$, as a function of $L$. The values $c$, $d$ and $T$ are constants. We assume that $L>0$ and $T>0$. $$ D(L) = \begin{cases} c \cdot L &\text{if}\ L \leq T \ \text{max}\left( D\left(\left\lfloor \frac L2 \right\rfloor \right), D\left(L - \left\lfloor \frac L2 \right\rfloor \right)\right) + d &\text{otherwise} \end{cases} $$ Below the threshold T, the algorithm proceeds sequentially and takes time c to process each single element. Above the threshold, the algorithm is applied recursively over the two halves of the array. The results are then merged using an operation that takes d units of time. Is it the case that for all $1 \leq L_1 \leq L_2$ we have $D(L_1) \leq D(L_2)$? If it is the case, prove the property by induction on $L$. If it is not the case, give a counterexample showing values of $L_1$, $L_2$, $c...	To determine whether the property ( D(L_1) \leq D(L_2) ) holds for all ( 1 \leq L_1 \leq L_2 ) with respect to the given depth function ( D(L) ), we need to analyze the behavior of the function based on its definition. ### Step 1: Understanding the Function Definition The function ( D(L) ) is defined piecewise: 1. Base Case: For ( L \leq T ), ( D(L) = c \cdot L ). This means that when the size of the array segment is small (i.e., less than or equal to the threshold ( T )), the depth is linear with respect to ( L ). 2. Recursive Case: For ( L > T ), the function is defined recursively as: [ D(L) = \max\left(D\left(\left\lfloor \frac L2 \right\rfloor\right), D\left(L - \left\lfloor \frac L2 \right\rfloor\right)\right) + d. ] This indicates that for larger sizes, the depth is determined by the maximum depth of the two halves of the array plus a constant ( d ) for the merging step. ### Step 2: Testing the Property We need to explore whether ...
  A service is an application component that performs long-running operations, usually in the background. A service doesn't provide a UI. While reviewing the pull request of a friend you notice that he periodically fetches data from the cloud in his activity? What potential problem this could lead to, and how can you fix it?	Executing a long-running operation like fetching data from the cloud directly in an activity can negatively impact the user experience, particularly if the network connection is slow, as it may cause the application to become unresponsive. To address this issue, he should implement a service to handle the data fetching. Within the service, it's important to spawn a new thread to perform the network operation, allowing the main thread to continue managing user interactions without interruption.
  Consider the problem of finding a maximum cardinality set packing in the semi-streaming model. An instance of this problem consists of a known universe $U$ of $n$ elements and sets $S \subseteq U$ are streamed one-by-one. The goal is to select a family $\mathcal{T}$ of pairwise disjoint sets (i.e., $S\cap S' = \emptyset$ for any two distinct sets $S, S' \in \mathcal{T}$) of maximum cardinality while only using $O(n\cdot \textrm{poly}\log n)$ storage space. Devise an algorithm in this setting that returns a set packing of cardinality at least $1/k$ times that of a maximum cardinality set packing, assuming that each streamed set $S$ has cardinality at most $k$, i.e., $	S

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 10
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.8
• Sentence Transformers: 4.1.0
• Transformers: 4.52.4
• PyTorch: 2.7.0+cu126
• Accelerate: 1.3.0
• Datasets: 3.6.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}