BGE m3 Uzbek Legal Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: uz
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("fitlemon/bge-m3-uz-legal-matryoshka")
# Run inference
sentences = [
    'O‘n olti yoshga to‘lguniga qadar nogironligi bo‘lgan bolani tarbiyalayotgan ota-onaga qanday qo‘shimcha kunlar beriladi, {chapter} va {section}da bu haqida nima yozilgan?',
    'Xodimga dam olish uchun emas, balki boshqa maqsadlarda beriladigan, xodimni mehnat \nmajburiyatlarini bajarishdan ozod etish davrlari dam olish vaqtiga kirmaydi. Bunday davrlar \njumlasiga quyidagilar kiradi:  \nmehnat shartnomasi ish beruvchining tashabbusiga k o‘ra bekor qilinishi to‘g‘risidagi \nogohlantirish muddati davrida xodimga ishga joylashish uchun beriladigan ishdan bo‘sh bo‘linadigan \nqo‘shimcha kunlar; \no‘n olti yoshga to‘lguniga qadar nogironligi bo‘lgan bolani tarbiyalayotgan ota -onadan \nbiriga (ota -onaning o‘rnini bosuvchi shaxsga) beriladigan ishdan bo‘sh bo‘linadigan qo‘shimcha \nkunlar; \nhomilador ayollarga beriladigan ishdan bo‘sh bo‘linadigan kunlar; \ndonorlarning tibbiy tekshiruv kunida hamda qon va uning tarkibiy qismlari topshiriladigan \nkunda ishdan ozod etilishi;  \nijtimoiy ta’tillar: homiladorlik va tug‘ish ta’tillari, bolani parvarishlash ta’tillari, o‘quv \nta’tillari va ijodiy ta’tillar; \nxodim tomonidan davlat yoki jamoat majburiyatlari bajariladigan davrlar; \nish beruvchining va mehnat jamoasining m anfaatlarini ko‘zlab majburiyatlar bajariladigan \ndavrlar; \nxodimning vaqtincha mehnatga qobiliyatsizlik davrlari; \nxodimga dam olish uchun emas, balki mehnat to‘g‘risidagi qonunchilikda va mehnat \nhaqidagi boshqa huquqiy hujjatlarda belgilangan o‘zga maqsadla rda beriladigan, xodimni mehnat \nmajburiyatlarini bajarishdan ozod etishning boshqa davrlari.',
    'Masofadan turib ishlovchi xodim bilan tuzilgan mehnat shartnomasi ushbu Kodeksda \nbelgilangan asoslarga ko‘ra bekor qilinishi mumkin. \nAgar masofadan turib ishlovchi xodimning ish beruvchining masofadan turib ishlash \nto‘g‘risidagi mehnat shartnomasini bekor qilish to‘g‘risidagi buyrug‘i bilan tanishib chiqishi elektron \nhujjat tarzida amalga oshirilsa, ish beruvchi masofadan turib ishlovchi xodimga mazkur mehnat \nshartnomasi bekor qilingan kuni lozim darajada rasmiylashtirilgan mehnat shartnomasini bekor qilish \nto‘g‘risidagi buyruqning ko‘chirma nusxasini ma’lum qilinadigan buyurtma xat bilan pochta orqali \nqog‘ozda yuborishi shart. \n4-§. Vaxta usulida ishlovchi shaxslarning mehnatini huquqiy jihatdan tartibga solishning \no‘ziga xos xususiyatlari',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: dim_1024, dim_768, dim_512, dim_256, dim_128 and dim_64
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

json

• Dataset: json
• Size: 4,737 training samples
• Columns: question and chunk
• Approximate statistics based on the first 1000 samples:

  	question	chunk
  type	string	string
  details	min: 9 tokens mean: 22.45 tokens max: 53 tokens	min: 26 tokens mean: 268.97 tokens max: 520 tokens

• Samples:

  question	chunk
  Ish beruvchi o‘rindoshlik asosida ishga qabul qilishda qanday hujjatlarni talab qilishga haqli emas?	Boshqa ish beruvchiga (asosiy ish joyidan tashqari) o‘rindoshlik asosida ishga kirayotgan shaxslar quyidagilarni taqdim etishi shart: pasportni yoxud uning o‘rnini bosuvchi hujjatni yoki identifikatsiya ID-kartasini; asosiy ish joyidan O ‘zbekiston Respublikasi Bandlik va mehnat munosabatlari vazirligi tomonidan tasdiqlanadigan shakl bo‘yicha ma’lumotnomani; bajarilishi uchun qonunchilikka muvofiq faqat muayyan ish stajiga ega bo‘lgan shaxslar qo‘yilishi mumkin bo‘lgan ishga o‘rindoshlik a sosida qabul qilishda asosiy ish joyidagi mehnat daftarchasining tasdiqlangan ko‘chirma nusxasini yoki elektron mehnat daftarchasidan ko‘chirmani; diplomni, guvohnomani (sertifikatni) yoki ta’lim to‘g‘risidagi yoki kasbiy tayyorgarlik haqidagi boshqa hujjatni, agar bunday ish maxsus bilimlarni yoxud maxsus tayyorgarlikni talab qilsa; mehnat sharoitlari zararli va (yoki) xavfli bo‘lgan ishga qabul qilish chog‘ida asosiy ish joyidan mehnatning xususiyati va shartlari to‘g‘risidagi olingan m...
  Yakka tartibdagi mehnatga oid munosabatlarni tartibga solishning asosiy jihatlari nimalardan iborat?	Yakka tartibdagi mehnatga oid munosabatlarni va ular bilan bevosita bog‘liq bo‘lgan ijtimoiy munosabatlarni huquqiy jihatdan tartibga solishning asosiy prinsiplari quyidagilardan iborat: mehnat huquqlarining tengligi, mehnat va mashg‘ulotlar sohasida kamsitishni taqiqlash; mehnat erkinligi va majburiy mehnatni taqiqlash; mehnat sohasidagi ijtimoiy sheriklik; mehnat huquqlari ta’minlanishining va mehnat majburiyatlari bajarilishining kafolatlanganligi; xodimning huquqiy holati yomonlashishiga yo‘l qo‘yilmasligi.
  Tashkilotning ta’sis hujjatlari ish beruvchining huquqlarini qanday ta'sir qiladi?	Ish beruvchi moddiy zarar yetkazilgan aniq sharoitlarni hisobga olgan holda zararni aybdor xodimdan to‘liq yoki qisman undirishdan voz kechish huquq iga ega. Tashkilot mulkdori ish beruvchining mazkur huquqini qonunchilikda, shuningdek tashkilotning ta’sis hujjatlarida nazarda tutilgan hollarda cheklashi mumkin.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• fp16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+cu124
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}