Model Card for kuntur-peru-legal-es-gemma-2b-it-merged ⚖️

The kuntur-peru-legal-es-gemma-2b-it-merged model is a state-of-the-art language model fine-tuned specifically for legal text comprehension and generation tasks in Spanish, focusing on the legal context of the Peruvian Constitution. Leveraging advanced techniques such as Low-Rank Adaptation (LoRA) and Bits and Bytes Quantization (BNB), this model provides accurate and contextually relevant responses to legal queries, making it a valuable tool for legal professionals, researchers, and AI enthusiasts interested in the legal domain.

Table of Contents

• kuntur-peru-legal-es-gemma-2b-it-merged ⚖️
  • Model Description 📘
  • Finetuning Progress 🛠️
    • Recommendations 📝
  • How to Get Started with the Model 🚀
  • Training Dataset 🧠
  • Finetuning Progress 🤖
  • • Environment and Libraries 🖥️
    • QLoRA Configuration 🧮
    • Model Merging and Saving 💾
  • Logging with Wandb 📊
  • Environmental impact 🌳

Model Description

kuntur-peru-legal-es-gemma-2b-it-merged is a contextual legal language model designed to provide personalized legal advice in Spanish based on Peruvian legal texts. Leveraging advanced techniques like LoRA, the model offers accurate and contextually relevant responses to legal queries, covering various aspects of Peruvian law and regulation. Whether it's understanding rights, navigating legal procedures, or interpreting statutes, kuntur-peru-legal-es-gemma-2b-it-merged empowers users with comprehensive and reliable legal guidance tailored to the Peruvian legal landscape.

• Developed by: Alonso Quispe
• Model type: Causal Language Model, specially fine-tuned with LoRA for the distinct domain of Peruvian law and regulation.
• Language(s) (NLP): Spanish, tailored for the legal and regulatory context of Peru.
• License: Apache License. This open-source license ensures that the model can be freely used, modified, and distributed. Please check the model's page on Hugging Face for specific licensing details.
• Base model: google/gemma-2b-it`

Recommendations

Users should verify model outputs against current regulations and consult with professionals for critical applications. Awareness of the model's scope and limitations is crucial for effective use.

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer

# Cargar el modelo y el tokenizador
model_name = "somosnlp/kuntur-peru-legal-es-gemma-2b-it-merged"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Ejemplo de uso
input_text = "Una persona ha sido despedida de su trabajo injustamente y necesita entender cuáles son sus derechos laborales según la Constitución Política del Perú."
input_ids = tokenizer.encode(input_text, return_tensors="pt")
output = model.generate(input_ids, max_length=100, num_return_sequences=1, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Dataset

The kuntur-peru-legal-es-gemma-2b-it-merged model was fine-tuned exclusively on the "Constitución Política del Perú dataset available through Hugging Face Datasets. This dataset serves as a rich source of questions and answers pertaining to the legal framework outlined in the Peruvian Constitution. The dataset encompasses a wide range of topics and provisions within the Peruvian Constitution, providing comprehensive coverage of constitutional principles, rights, and legal interpretations.

Train Progress

Eval Progress

---

Environment and Libraries

The training process was executed within a Python environment, utilizing essential libraries to facilitate various tasks:

• transformers: Used for loading and fine-tuning the model, providing a seamless interface for working with pre-trained language models.
• datasets: Employed for efficient handling and preprocessing of the training dataset, ensuring streamlined data pipelines.
• torch: Utilized as the primary deep learning framework to support model training and optimization.
• peft: Integrated for applying Low-Rank Adaptation (LoRA) techniques to the model, enabling efficient adaptation to specialized domains without compromising performance.
• qlora: Additionally utilized for further model adaptation, enhancing its ability to comprehend and generate responses specific to the legal context of the "Constitucióm Política del Perú" dataset.

QLoRA Configuration

QLoRA (Quantization LoRA) was employed to optimize the model's computational efficiency and memory footprint while preserving its accuracy. Two configurations were utilized:

• BitsAndBytesConfig: This configuration enabled the model to load in 4-bit quantization, leveraging the nf4 quantization type with a torch.bfloat16 compute data type for enhanced efficiency during inference.

      bnb_config = BitsAndBytesConfig(
      load_in_4bit=True,
      bnb_4bit_quant_type="nf4",
      bnb_4bit_compute_dtype=torch.bfloat16,
      bnb_4bit_use_double_quant=False,
  )
  

• LoRAConfig: This configuration applied Low-Rank Adaptation (LoRA) to the model, optimizing its parameters for the specific task of language modeling in the legal domain. Key parameters include:

  • r: Reduced to 8 from the default 32, controlling the rank of adaptation layers.
  • lora_alpha: Adjusted to 16 from the default 64, regulating the sparsity of adapted weights.
  • target_modules: Specified modules for adaptation, focusing on query, key, value, and output projection layers.
  • bias: Set to "none" to exclude bias terms from adaptation, simplifying the model architecture.
  • lora_dropout: Reduced to 0.025 from the default 0.05, controlling the dropout rate during adaptation.
  • task_type: Configured as "CAUSAL_LM" to indicate the task type of the language model.

      config = LoraConfig(
      r=8,
      lora_alpha=16,
      target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj'],
      bias="none",
      lora_dropout=0.025,
      task_type="CAUSAL_LM",
  )
  

These configurations were crucial for optimizing the model's performance and resource utilization during training and inference, ensuring efficient deployment.

Model Merging and Saving

After fine-tuning, the LoRA-adjusted weights were merged back with the base Gemma model to create the final kuntur-peru-legal-es-gemma-2b-it-merged. The model was then saved and made available through Hugging Face for easy access and further development.

Logging with Wandb

During the training process, Wandb (Weights & Biases) was used for comprehensive logging and visualization of key metrics. Wandb's powerful tracking capabilities enabled real-time monitoring of training progress, evaluation metrics, and model performance. Through interactive dashboards and visualizations, Wandb facilitated deep insights into the training dynamics, allowing for efficient model optimization and debugging. This logging integration with Wandb enhances transparency, reproducibility, and collaboration among researchers and practitioners.

• eval/loss:1.1386919021606443
• eval/runtime:44.2153
• eval/samples_per_second:8.707
• eval/steps_per_second:8.707
• train/epoch:49.62
• train/global_step:4,850
• train/grad_norm:3.5548949241638184
• train/learning_rate:0
• train/loss:0.8596
• train/total_flos:236,149,029,419,876,350
• train/train_loss:1.105836234535139
• train/train_runtime:13,237.4947
• train/train_samples_per_second:5.9
• train/train_steps_per_second:0.366

Environmental impact

The training of kuntur-peru-legal-es-gemma-2b-it-merged was conducted optimizing the computational expenditure required.

• Hardware Type: NVIDIA A10G GPU
• Hours Utilized: Approximately 4 hours
• Energy Consumption: Approximately 300 Wh
• Estimated CO2 Emissions: Approximately 368.75g

To-Do List

Dataset Generation, Human Feedback Review with Argilla, and Finetuning with the Following Legal Texts:

License

This project is distributed under the Apache 2.0 license.