distilbert-base-uncased finetuned on MSMARCO with MarginMSELoss + MultipleNegativesRankingLoss

This is a sentence-transformers model finetuned from distilbert/distilbert-base-uncased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 300 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Dot Product
• Supported Modality: Text

• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kwondw/distilbert-base-uncased-msmarco-margin-mse-mnrl-502939")
# Run inference
sentences = [
    'The weather is lovely today.',
    "It's so sunny outside!",
    'He drove to the stadium.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact, NanoTouche2020, NanoMSMARCO, NanoNFCorpus and NanoNQ
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Training Details

Training Dataset

Unnamed Dataset

• Size: 502,939 training samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, negative_7, negative_8, negative_9, negative_10, and label
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	negative_9	negative_10	label
  type	string	string	string	string	string	string	string	string	string	string	string	string	list
  details	min: 4 tokens mean: 9.12 tokens max: 28 tokens	min: 19 tokens mean: 81.57 tokens max: 215 tokens	min: 18 tokens mean: 79.85 tokens max: 238 tokens	min: 19 tokens mean: 79.86 tokens max: 213 tokens	min: 17 tokens mean: 78.78 tokens max: 212 tokens	min: 16 tokens mean: 79.86 tokens max: 228 tokens	min: 16 tokens mean: 79.48 tokens max: 217 tokens	min: 22 tokens mean: 80.47 tokens max: 223 tokens	min: 18 tokens mean: 79.14 tokens max: 207 tokens	min: 20 tokens mean: 80.19 tokens max: 213 tokens	min: 20 tokens mean: 79.38 tokens max: 225 tokens	min: 17 tokens mean: 80.5 tokens max: 263 tokens	size: 10 elements

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	negative_9	negative_10	label
  what county is santa rosa, ca	Sponsored Topics. Santa Rosa is the county seat of Sonoma County, California, United States. The 2010 census reported a population of 167,815. Santa Rosa is the largest city in California's Wine Country and fifth largest city in the San Francisco Bay Area, after San Jose, San Francisco, Oakland, and Fremont and 26th largest city in California.	Santa Rosa, CA is in the western part of California in the pacific United States. Closest County to Santa Rosa, CA: Sonoma County. Neighboring Cities to Santa Rosa, CA: Larkfield-Wikiup, Sebastopol, Roseland, Fulton. ZIP Codes for Santa Rosa, CA: 95405, 95404, 95407, 95401, 95403, 95409, 95472. 3-Digit ZIP Code Prefix for Santa Rosa, CA:	Location Overview. ​The Santa Rosa Area office of the California Highway Patrol provides service to the majority of Sonoma County, (the Sonoma Valley east of Trinity Rd. is serviced by the Napa Area office of the CHP.) The Santa Rosa Area CHP provides several programs to achieve our public safety goals. Click on Programs and Services above for more information.	The Laguna de Santa Rosa Foundation says: The Laguna de Santa Rosa is Sonoma County's richest area of wildlife habitat, and the most biologically diverse region of Sonoma County (itself the second-most biologically diverse county in California)...	Santa Rosa Metropolitan Statistical Area (MSA) (Sonoma County) Contributor: California. Labor Market Information Division: Publisher: Labor Market Information Division, 1996 : Export Citation: BiBTeX EndNote RefMan	Santa Rosa, CA Real Estate Insights Santa Rosa is a city in and the county seat of Sonoma County, California. You will never be hungry living in Santa Rosa, since you only have to walk a couple of blocks to find food from all over the world, such as pizza, French and Mexican.	Santa Rosa (Sun-tah Rousa), is the county seat of Guadalupe County, N. Mexico. The Pecos River valley has been inhabited for over 10,000 years (the nearby archaeological site at Clovis NM has some of the oldest stone tools in America).	Santa Rosa is home to the museum that celebrates famed cartoonist Charles M. Schulz’s life and work. Note the colorful street sculptures of Peanuts characters throughout the city. On the museum’s campus is Snoopy’s Home Ice, also known as the Redwood Empire Ice Arena. Adjacent is the new Children's Museum of Sonoma County.	Photo of Charles M Schulz Sonoma County Airport - STS - Santa Rosa, CA, United States by Werner V.	Spring Lake Regional Park in Santa Rosa is one of Sonoma County's most popular parks, featuring trails, a campground, picnic areas, a natural history center, and a summer swimming lagoon. Trails. With nearly 10 miles of trails, Spring Lake is a favorite destination for hiking, running, biking and horseback riding.	LMI for Santa Rosa MSA, California The files listed below are for the Santa Rosa-Petaluma Metropolitan Statistical Area (MSA), California, which is comprised of Sonoma County. Questions about these data and the local area economy may be addressed by contacting a Local Labor Market Consultant.	[1.187312126159668, 2.719836711883545, 3.025529384613037, 2.8301777839660645, 0.28928375244140625, ...]
  how to check a vacuum leak	There are many problems that vacuum leaks can cause with your car. Vacuum leaks can cause your car to have a fast idle speed and to hesitate, or even misfire, when accelerating. This guide will go over two ways to check for vacuum leaks on a car engine.	An intake manifold gasket can be the source of a vacuum leak and/or engine coolant leak. An intake gasket with a vacuum leak will often have a rough idle and engine misfires.This may turn on the Check Engine Light and set misfire and/or fuel mixture out of range trouble codes.Some intake manifolds are made of plastic. They can become pitted and may need to be replaced along with the intake gaskets.n intake manifold gasket can be the source of a vacuum leak and/or engine coolant leak. An intake gasket with a vacuum leak will often have a rough idle and engine misfires.	Visit me at: http://www.ericthecarguy.com/. Here's another great use for your vacuum gauge. If you're trying to find TDC (Top Dead Center) compression stroke say for a leak down test, this is an 'easy' way to do that. Also, if you are doing a leak down test you're already set up to do the test once you find TDC.This method could save you quite a bit of time.f you're trying to find TDC (Top Dead Center) compression stroke say for a leak down test, this is an 'easy' way to do that. Also, if you are doing a leak down test you're already set up to do the test once you find TDC.	There are many problems that vacuum leaks can cause with your car. Vacuum leaks can cause your car to have a fast idle speed and to hesitate, or even misfire, when accelerating. This guide will go over two ways to check for vacuum leaks on a car engine.	Never use a vacuum to test for leaks. You will not be able to find a leak under vacuum, and you will contaminate the system with moisture and noncondensible gases. Never pull a vacuum through a Schrader valve. This practice will only increase your evacuation time due to the high restriction it will cause.	If the vacuum level at the check valve is 18”, check that the booster check valve is working. Disconnect the vacuum hose at the check valve and attach a piece of tubing. Blow into the valve. If air passes through, the valve is defective and must be replaced.	The first thing to check for here is a vacuum leak. Sometimes it can be difficult to track down broken hoses, but it takes a large vacuum leak to cause a high idle. Check the rubber hoses that connect directly to the intake manifold and throttle body. These are the most likely to fail.	Vacuum leaks can be located with spray carburetor cleaner or a can of WD-40. If the area is obstructed by linkage or hoses, use an extension nozzle to pinpoint the area of the vacuum leak.	The EVAP leak check monitor relies on the individual components of the enhanced EVAP system to either allow a natural vacuum to occur in the fuel tank or apply engine vacuum to the fuel tank and then seal the entire enhanced EVAP system from the atmosphere.. The FTP sensor is used by the engine on EVAP leak check monitor to determine if the target vacuum necessary to carry out the leak check on the fuel tank is reached. Some vehicle applications with the engine on EVAP leak check monitor use a remote in-line FTP sensor.	Then the pipe shall be inspected. This test is usually completed by the client when the system is put into service by the client. Section 14 Leak Testing 14-2 Piping/Mechanical Handbook 1996:Rev.2 Vacuum Leak Testing This is the hardest type of test with which to find a leak.	If you have checked for leaks between the solenoid and the hubs,and you have no loss, it will probably be the solenoid. But make sure you check vacuum with a gauge coming to the solenoid while also playing with other vacuum operated funtions. You may have a loss in vacuum from another component.	[0.584254264831543, 1.3918685913085938, 0.0, -0.4763789176940918, 1.3648402690887451, ...]
  most expensive cities in the world	Singapore has retained its position as the world's most expensive city, according to research by the Economist Intelligence Unit (EIU). The top five most expensive cities in the world remain unchanged from a year earlier and include, in descending order, Paris, Oslo, Zurich and Sydney.	Seoul, South Korea, one of the many Asian cities on the list.Creative RF / Getty Images. The investment consultancy firm Mercer has ranked the world's most expensive cities to live in as part of its annual Cost of Living Survey. Contrary to the assumption that Western cities are the costliest, Southeast Asia and Africa feature heavily in the top 19, continuing the trend of emerging markets adjusting to massive outside investment.	Of the world's most expensive cities, Singapore has kept its title for the third year running. The Asian island city has beat out Zurich and Hong Kong to top the latest rankings by the Economist Intelligence Unit, which surveys 400 individual prices across 160 products and services, and then compares all places with the base city of New York.	In that case, these places should not be on their list. The most expensive cities in the world are mostly in Asia and Western Europe, according to the Economist Intelligence Unit's Worldwide Cost of Living survey.	Some of the findings come as no surprise; like the fact that New York City is #1 on the list for the overall price index–the relative cost of goods and services worldwide. Remove rent from the price index, and Zurich and Geneva top the list of world’s most costly cities ahead of New York. World's Most Expensive Cities.	Singapore tops the list of the most expensive city in the world once again, a new survey comparing the cost of living in 133 cities reveals. Nudging just behind Singapore is Paris, followed by Oslo, Zurich and Sydney. London is the world's eleventh most expensive city, The Economists' Intelligence Unit report says. London is now as expensive as Tokyo, which was replaced by Singapore as the most expensive city last year. Cities in Pakistan and India, including Karachi, Bangalore and Mumbai make up four out of the top five cheapest cities to live in last year.	London, United Kingdom. Having been rated as the world’s most expensive city to visit for 24 hours it should come as no surprise that London would be on this list somewhere. What may be surprising to some of you, however, is that there are at least 24 cities even more expensive. 24. Brisbane, Australia. Although it is not Australia’s most expensive city it does arguably have one of the world’s most expensive public transportation systems (trailing only London and Oslo) 23. Canberra, Australia. Moving from one Australian city to another this time we’re in the capital of the land down under.	March 7, 2017. Once again, international design and consultancy firm Arcadis has named New York City as the world's most expensive city in which to build, according to its annual International Construction Costs Index.	But while Asian cities dominated the list of the top 50 most expensive cities in the world (clinching 26 of the 50 slots), the continent also housed some of the least expensive cities. Tokyo remains the most expensive location in Asia-Pacific region (8th globally), followed closely by Hong Kong (9th globally).	Find out in the slideshow below. See how these locales compare to the most expensive cities to travel to around the world and the world's most expensive cities for hotel rooms. Or, there are always the best value cities in Europe for budget travelers. Most Expensive Monuments.	The Southeast Asian city-state of Singapore retained its title as the world's most expensive city for the second consecutive year, the Economist Intelligence Unit (EIU) said in a new survey.	[1.1380739212036133, 0.32665157318115234, 1.4172601699829102, 0.9240751266479492, -0.14728260040283203, ...]

• Loss: main.CombinedMarginMSEMNRLLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 24
• per_device_eval_batch_size: 24
• num_train_epochs: 1
• warmup_steps: 0.1
• fp16: True

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 24
• per_device_eval_batch_size: 24
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Training Time

• Training: 3.7 hours
• Evaluation: 2.4 minutes
• Total: 3.7 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}