Writers

"Less Gemma - more human"

Overview

I had a lot of fresh and creative experience with both ConicCat tunes, but they had some instability issues, that I hopefully fixed there.

Mergekit Configuration

Phase 1:

della_linear Recipe 1

merge_method: della_linear
base_model: F:\AI\Merge\Gemma-4-it
tokenizer_source: base
dtype: bfloat16
parameters:
  lambda: 1.0
  epsilon: 0.05
  normalize: true
models:
  - model: F:\AI\Merge\Gemma-4-Writer-D
    parameters:
      density: 0.30
      weight:         
        - {filter: mlp.gate_proj, value: 0.25}
        - {filter: mlp.up_proj, value: 0.25}
        - {filter: mlp.down_proj, value: 0.25}
        - {value: 0.40}


  - model: F:\AI\Merge\Gemma-4-Writer-F
    parameters:
      density: 0.40
      weight:
        - {filter: mlp.gate_proj, value: 0.30}
        - {filter: mlp.up_proj, value: 0.30}
        - {filter: mlp.down_proj, value: 0.25}
        - {value: 0.40}

  - model: F:\AI\Merge\Gemma-4-Scotoma-V2
    parameters:
      density: 0.30
      weight:
        - {filter: mlp.gate_proj, value: 0.20}
        - {filter: mlp.up_proj, value: 0.20}
        - {filter: mlp.down_proj, value: 0.40}
        - {value: 0.20}

Special Thanks

• Google DeepMind: Gemma.
• The Open-Source Community: People that keep working and sharing their thoughts.
• Mergekit Fork: Zerofata - merges is an addiction.
• Tunes: ConicCat for making these work and breathing new life into Gemma.
• To Nimbz: Cat that keeps me on my toes.