🌅 MN Aura 12B v1

This is a merge of pre-trained language models created using mergekit-exp.

Merge Details

Merge Method

This model was merged using the Adaptive Unified Riemannian Annealing (AURA) merge method using Retreatcost/Mistral-Nemo-Base-2407-ChatML as a base.

Models Merged

The following models were included in the merge:

• DreadPoor/Famino-12B-Model_Stock
• EldritchLabs/MN-Starlight-Sylph-12B
• OccultAI/MN-Nazgul-12B-v1
• Retreatcost/Mistral-Nemo-Base-2407-ChatML
• shrugging-shoulders/Amberlight-Lux-12B
• WokeAI/Tankie-DPE-12B-SFT-v2

Configuration

The following YAML configuration was used to produce this model:

architecture: MistralForCausalLM
base_model: B:\12B\Retreatcost--Mistral-Nemo-Base-2407-ChatML
models:
  - model: B:\12B\DreadPoor--Famino-12B-Model_Stock
    parameters:
      weight: 0.4
      density: 0.90
      epsilon: 0.09
      pinocchio: false
  - model: B:\12B\EldritchLabs--MN-Starlight-Sylph-12B
    parameters:
      weight: 0.4
      density: 0.90
      epsilon: 0.09
      pinocchio: false
  - model: B:\12B\shrugging-shoulders--Amberlight-Lux-12B
    parameters:
      weight: 0.4
      density: 0.90
      epsilon: 0.09
      pinocchio: false
  - model: B:\12B\WokeAI--Tankie-DPE-12B-SFT-v2
    parameters:
      weight: 0.4
      density: 0.90
      epsilon: 0.09
      pinocchio: false
  - model: B:\12B\OccultAI--MN-Nazgul-12B-v1
    parameters:
      weight: 0.4
      density: 0.90
      epsilon: 0.09
      pinocchio: true
merge_method: aura # v6
parameters:
  tol: 1.0e-11
  max_iter: 25
  eta: 1.25
  theta: 0.93
  optimization_mode: "hybrid"
  phi: 2.0
  alpha: 0.25
  kappa: 0.75
  jl_projs: 8
  halley_strength: 1.61803
  halley_period: 8
  scalar_gravity: -0.025
  mu: 0.5
  aura_shift: true
  omega: 0.08
  momentum: 0.25
  patience: 64
  sharp: 0.5
  voice: 0.25
  density: 0.9
  epsilon: 0.09
  rescale: true
  int8_mask: false
  normalize_weights: false
  normalize_router: true
  router_strategy: "della"
  blend_experts: true
  auto_pad: true
  skip_on_mismatch: true
  verbose_logging: true
  resume_path: "B:/mergekit-new/Aura_Cache"
  enable_qliphoth: true
  pinocchio_lambda: 0.75
  inverse_pinocchio: false
  dynamic_pareto_pinocchio: true
  qliphoth_gamma: 0.5
  qliphoth_lambda_enslaved: 0.5
  qliphoth_svd_rank: 0
  qliphoth_preserve_ratio: 1.0
  save_layer_heatmaps: false
  audit_image_dir: "B:/mergekit-new/Aura_Heatmaps"
  gap_jumper: 1.0
  enable_qhe: true
  qhe_levels: 16
  qhe_intensity: 0.3
  qhe_crossover: 0.15
  coherence_override: 1.0
  conflict_scale_override: 1.0
  invert_coherence_to_chaos: true
dtype: float32
out_dtype: bfloat16
tokenizer:
  source: union
chat_template: auto