CosmeticVenture
This is a specialized model merge created using MergeKit, combining the French language expertise of Mistral-7B with the business acumen of Zephyr-7B through an advanced SLERP fusion technique with variable interpolation values, optimized for the cosmetic industry.
About Me
I'm David Soeiro-Vuong, a third-year Computer Science student working as an apprentice at TW3 Partners, a company specialized in Generative AI. Passionate about artificial intelligence and language models optimization, I focus on creating efficient model merges that balance performance and capabilities.
Project Overview
CosmeticVenture is designed as a specialized LLM for the cosmetic industry, capable of providing expert business guidance for entrepreneurs and professionals in this sector. The model combines Mistral's excellent French language capabilities with Zephyr's business understanding to create a powerful advisor for cosmetic business development.
Merge Details
Merge Method
This model uses SLERP (Spherical Linear Interpolation) with carefully calibrated parameters:
- Attention Layers: Variable interpolation values [0, 0.5, 0.3, 0.7, 1] leveraging Zephyr's instruction-following and business capabilities
- MLP Layers: Variable interpolation values [1, 0.5, 0.7, 0.3, 0] maintaining Mistral's French language expertise and reasoning
- Other Parameters: 0.5 interpolation value creating a balanced fusion
- Format: bfloat16 precision for efficient memory usage
Models Merged
- mistralai/Mistral-7B-v0.1 - A powerful French-origin base model with excellent multilingual capabilities
- HuggingFaceH4/zephyr-7b-beta - An instruction-tuned model with strong business reasoning capabilities
Configuration
slices:
- sources:
- model: mistralai/Mistral-7B-v0.1
layer_range: [0, 32]
- model: HuggingFaceH4/zephyr-7b-beta
layer_range: [0, 32]
merge_method: slerp
base_model: mistralai/Mistral-7B-v0.1
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5
dtype: bfloat16
MODEL_NAME: "CosmeticVenture"
Model Capabilities
This specialized merge combines:
- Mistral's excellent French language understanding and multilingual capabilities
- Zephyr's instruction-following and business reasoning abilities
- Domain adaptation for the cosmetic industry through strategic parameter fusion
The resulting model is optimized for tasks in the cosmetic business sector, such as:
- Regulatory guidance for cosmetic products in various markets
- Business planning and market analysis for cosmetic startups
- Ingredient formulation and technical documentation assistance
- Marketing strategy and brand positioning advice
- Trend analysis and innovation forecasting for beauty products
Future Development
This interim model will be further enhanced once access to Meta-Llama-3-8B is granted, providing even stronger business reasoning capabilities. The model will also be fine-tuned on a specialized dataset consisting of:
- Regulatory documentation (EU regulations, BPF, ANSM guidelines)
- Market research and industry analyses (FEBEA reports, segment studies)
- Technical documentation (ingredient specifications, manufacturing processes)
- Business resources (business plans, pricing strategies, case studies)
- Industry trends (clean beauty, solid cosmetics, circular economy)
- Marketing strategies (brand positioning, digital strategies)
Limitations
- Limited domain-specific training beyond parameter merging
- May require additional fine-tuning for highly specialized cosmetic industry tasks
- General limitations inherited from the base 7B parameter models
- Pending enhancement with Meta-Llama-3-8B once access is granted
License
This model is released under the Apache 2.0 license, consistent with the underlying models' licenses.