SentenceTransformer based on google-bert/bert-base-uncased

This is a sentence-transformers model finetuned from google-bert/bert-base-uncased. It maps inputs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google-bert/bert-base-uncased
• Maximum Sequence Length: 32 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'subaru -',
    'subaru impreza - petrol 2000',
    'honda br - v s diesel 1498',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9939, 0.9908],
#         [0.9939, 1.0000, 0.9881],
#         [0.9908, 0.9881, 1.0000]])

Evaluation

Metrics

Triplet

• Dataset: fw-variant-dev
• Evaluated with TripletEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 86,450 training samples
• Columns: damaged and original
• Approximate statistics based on the first 100 samples:

  	damaged	original
  type	string	string
  modality	text	text
  details	min: 4 tokens mean: 11.63 tokens max: 22 tokens	min: 8 tokens mean: 16.18 tokens max: 25 tokens

• Samples:

  damaged	original
  1.5 petrol 1482	hyundai alcazar 1.5 executive mt petrol 1482
  mahindra. 300 1.5 w8 tone diesel 1497	mahindra. xuv 300 1.5 w8 dual tone diesel 1497
  mahindra scorpio vlx mhawk bs iv airbag diesel 2179	mahindra scorpio vlx 2.2 mhawk bs iv airbag diesel 2179

• Loss: DenoisingAutoEncoderLoss with these parameters:

  {
      "decoder_name_or_path": "google-bert/bert-base-uncased",
      "need_retokenization": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 1,765 evaluation samples
• Columns: damaged and original
• Approximate statistics based on the first 100 samples:

  	damaged	original
  type	string	string
  modality	text	text
  details	min: 5 tokens mean: 11.19 tokens max: 20 tokens	min: 9 tokens mean: 15.5 tokens max: 29 tokens

• Samples:

  damaged	original
  bentley continental gt 6.0 speed convertible 4 petrol	bentley continental gt 6.0 speed convertible 4 seater petrol 5998
  tata 1.2 rtn bs-vi 1199	tata altroz xt 1.2 rtn bs-vi petrol 1199
  maruti s-cross hybrid diesel 1248	maruti s-cross hybrid diesel 1248

• Loss: DenoisingAutoEncoderLoss with these parameters:

  {
      "decoder_name_or_path": "google-bert/bert-base-uncased",
      "need_retokenization": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• num_train_epochs: 10
• learning_rate: 1e-05
• warmup_steps: 0.1
• bf16: True
• per_device_eval_batch_size: 64
• seed: 46

All Hyperparameters

Click to expand

• per_device_train_batch_size: 64
• num_train_epochs: 10
• max_steps: -1
• learning_rate: 1e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 64
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 46
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Click to expand

Training Time

• Training: 38.6 minutes

Framework Versions

• Python: 3.12.0
• Sentence Transformers: 6.1.0
• Transformers: 5.20.1
• PyTorch: 2.14.1+cu130
• Accelerate: 1.15.0
• Datasets: 5.1.0
• Tokenizers: 0.23.3

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

DenoisingAutoEncoderLoss

@inproceedings{wang-2021-TSDAE,
    title = "TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning",
    author = "Wang, Kexin and Reimers, Nils and Gurevych, Iryna",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    pages = "671--688",
    url = "https://arxiv.org/abs/2104.06979",
}