nomic-embed text-v1.5 Clinical-Trials Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/nomic-embed-text-v1.5 on the dataset from Clinical_trials_anchor-positive-pairs_EmbeddingModel-data_final. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/nomic-embed-text-v1.5
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset: Clinical_trials_anchor-positive-pairs_EmbeddingModel-data_final
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'NomicBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("vab46/nomic-embed-text-v1.5_Clinical-Trials_Matryoshka_final")
# Run inference
documents = [
    'TITLE: Safety and Efficacy of Six-Channel Radiofrequency Ablation System for Renal Denervation in Patients With Untreated Grade I Hypertension: a Pilot Study\nSUMMARY: Prospective, Multi-Center, Randomized, shame-Controlled, Uptake clinical trial to evaluate the efficacy and safety of the six-channel radio-frequency（RF) renal denervation system-comprising the six-channel RF generator (specification model: 25D1G, software release version: SRG-V1) and the disposable ultra-guiding RF denervation catheter (specification model: 25C6W127F115T)-for renal denervation in patients with grade I hypertension and without taking antihypertensive medicines.\nINCLUSION_CRITERIA: 1. Male or female, aged 18 to 65 years inclusive\n2. Hypertension duration longer than 3 months\n3. Hypertensive subjects who have been stopped taking antihypertensive drugs continuously and stably for at least 4 weeks or who do not take antihypertensive drugs , with office systolic/diastolic blood pressure still ≥140/90 mmHg and \\<160/100 mmHg, and 24-hour ambulatory mean systolic /diastolic pressure ≥130/80 mmHg and \\<140/90 mmHg;\n4. The subject or his/her legal representative fully understands the content of the informed consent form for this trial and voluntarily signs the written informed consent form.',
]
queries = [
    "I have high blood pressure, but I'm not on any medication. Can I participate in this study?",
    'What is the primary goal of this clinical trial regarding pain management in premature newborns?',
    "I'm on a ventilator and feel really short of breath. Can anything be done to make me feel more comfortable?",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5783, 0.0218, 0.0940]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768, dim_512, dim_256, dim_128, dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

Comparison with the baseline model performance on same data

For the list of Sequential Evaluator score on nested mashtroyska embeddings [768, 512, 256, 128, 64]:-

Initial(pre-trained) score:- 0.405----------->Final(fine-tuned) score:- 0.56

Training Details

Training Dataset

Unnamed Dataset

• Size: 7,079 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 100 samples:

  	positive	anchor
  type	string	string
  modality	text	text
  details	min: 72 tokens mean: 290.18 tokens max: 675 tokens	min: 11 tokens mean: 21.08 tokens max: 42 tokens

• Samples:

  positive	anchor
  TITLE: Risk Classification and Prediction of Histopathological Subtypes in Basal Cell Carcinoma Using a CNN-Based Artificial Intelligence Model on Dermoscopic Images SUMMARY: This retrospective observational study aims to develop and evaluate a convolutional neural network (CNN)-based artificial intelligence model for risk classification and histopathological subtype prediction of basal cell carcinoma (BCC) using clinical and dermoscopic images. Histopathologically confirmed BCC cases from a dermatology archive will be included. The primary objective is to assess the diagnostic performance of the CNN model in classifying BCC as low-risk or high-risk. Secondary objectives include predicting histopathological subtypes and comparing the model's performance with that of dermatology physicians. Histopathological diagnosis will serve as the reference standard. All archived data will be anonymized before analysis. INCLUSION_CRITERIA: Patients with histopathologically confirmed basal cell carc...	Are dermoscopic images with sufficient image quality and resolution required for participation in this trial?
  TITLE: Postural Control Mechanism During a Stationary Wheelchair Wheelie SUMMARY: The purpose of this study was to investigate the postural control during a stationary wheelchair wheelie. A group of participants was recruited for this observational study. Participants were asked to perform and maintain the stationary phase of a wheelchair wheelie, during which kinematic and kinetic data were recorded. INCLUSION_CRITERIA: Spinal cord injury at or below the T1 (first thoracic) level. Between 20 and 65 years of age. Currently using a manual wheelchair as the primary means of mobility. Able to perform and maintain a wheelchair wheelie for at least 20 seconds	Is there an age restriction for participants in this clinical trial, and if so, what is it?
  TITLE: The Effect of Mandala Art Therapy on Fear and Anxiety of Childbirth in Expectant Fathers Waiting Outside the Delivery Room: A Randomized Controlled Study SUMMARY: This randomized controlled trial is designed to evaluate the effectiveness of mandala art therapy in reducing childbirth fear and state anxiety among fathers whose partners are nulliparous pregnant women and who are waiting in the obstetric ward waiting area during labor. Eligible participants will be randomly allocated to either the intervention or the control group. Fathers assigned to the intervention group will participate in a single 20-25-minute mandala art therapy session, whereas those in the control group will receive standard care without any additional intervention. Childbirth fear and state anxiety will be assessed using the Fathers' Birth Fear Scale and the State subscale of the Spielberger State-Trait Anxiety Inventory, respectively, at baseline and immediately following the intervention. The findings of ...	Could a father with a partner expecting their first child qualify for this study?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 4
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 0.1
• gradient_accumulation_steps: 8
• fp16: True
• per_device_eval_batch_size: 32
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 8
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 8
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 31.6 minutes

Framework Versions

• Python: 3.13.15
• Sentence Transformers: 5.7.0
• Transformers: 5.15.0
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}