Model Card for Model ID
This model aims to be a high-performance chatbot. During training, examples that have a quality score of less than 0.03 are skipped.
Model Details
Model Description
This model is to be used as a general-purpose chatbot/assistant. Trained on about 400,000 examples of M4-ai/Rhino, examples with a quality score lower than 0.03 are removed. During validation, this model achieved a loss of 0.55
This model was trained on the ChatML prompt format.
- Developed by: Locutusque
- Model type: mistral
- Language(s) (NLP): English
- License: cc-by-nc-4.0
- Finetuned from model: mistralai/Mistral-7B-v0.1
Uses
This model is to be used as a general-purpose assistant, and may need to be further fine-tuned on DPO to detoxify the model or SFT for a more specific task.
Direct Use
This model should be used as a general assistant. This model is capable of writing code, answering questions, and following instructions.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
Training Details
Training Hyperparameters
- Training regime: bf16 non-mixed precision
Evaluation
Testing Data, Factors & Metrics
Testing Data
First 100 examples of M4-ai/Rhino. Training data does not include these examples.
Results
Test loss - 0.48
Summary
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: 8 TPU V3s
- Hours used: 7
- Cloud Provider: Kaggle
- Compute Region: [More Information Needed]
- Carbon Emitted: 8.88