Gemma 4 12b Blorbo v0b

alternatively Gemma 4 12Blorbo

The mathematical average of every blorbo, circa 2014. Generated by Z-Image base.

UPDATE: New version! Now with actually untied embeddings and lm_head. Don't think it worked doe

A very scrunkly Gemma 4 12b finetune. I don't know why but its just got that scrimbo in it.

Mostly trained on:

• Reasoning data generated with Mimo v2.5 Pro, Mimo v2 Pro, and Doubao Seed 2.0 Pro; albeit Seed's reasoning was masked out when training, so it only learnt to reason in Mimo's style (which is typically shorter than Gemma's reasoning)
• Ye olde human roleplaying and story data

Shockingly, it seems to have learnt both mostly consistently!

May or may not have been trained correctly, update maybe soon :3

Usage

• Format: Regular Gemma 4, no changes needed. <|think|> should still work as normal to trigger reasoning.
• Samplers: Not sure, whatever you like? It worked fine for me at 1.0-1.25 temp and 0.1 min_p or 0.95 top_p

Credits

• Everyone at Allura for being cool and based
• Everyone who's data was trained on and everyone who collated the data trained on
• The global proletariat

Nerd details

Dataset/Pretokenization config

datasets:
  - path: allura-forge/mimo-v2.5-pro-distill-combined
    conversation_field: messages
    type: dan-chat-advanced
    reasoning_loss: true
    train_last_response_only: true
    prepend_system_prompt: "<|think|>"

  - path: allura-forge/mimo-v2-pro-claude-distill-hs3
    conversation_field: messages
    type: dan-chat-advanced
    reasoning_loss: true
    train_last_response_only: true
    system_prompt: "<|think|>The assistant is Claude."

  - path: allura-forge/doubao-seed2.0-claude-distill-hs3
    conversation_field: messages
    type: dan-chat-advanced
    reasoning_loss: false
    train_last_response_only: true
    system_prompt: "<|think|>The assistant is Claude."

  - path: estrogen/random-slop-dedup # pippa, limarp, random dan sets
    type: dan-chat-advanced
    reasoning_loss: false
    train_last_response_only: true
  - path: "redacted private set"
    type: dan-chat-advanced
    reasoning_loss: false
    train_last_response_only: true
  
  - path: ToastyPigeon/kimi-stories-instruct
    type: dan-chat-advanced
    conversation_field: messages
    reasoning_loss: false
    train_last_response_only: true

Training details & config

R64/A512 16 bit LoRA (equivalent to R64/A64 rsLoRA), optimized with ScheduleFree AdamW; the embedding layers were untied and both FFTed separately.

# === Model Configuration ===
base_model: allura-forge/gemma-4-12B-it-untied
load_in_8bit: false
load_in_4bit: false


# === Training Setup ===
num_epochs: 2
micro_batch_size: 1
gradient_accumulation_steps: 16
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

# === Hyperparameter Configuration ===
learning_rate: 1e-5
embedding_lr: 1e-6
lr_scheduler: constant_with_warmup
weight_decay: 0.0
max_grad_norm: 0.1
warmup_ratio: 0.05

optimizer: schedule_free_adamw

adapter: lora
lora_r: 64
lora_alpha: 512
lora_target_modules: 'model.language_model.layers.[\d]+.(_checkpoint_wrapped_module.)?(mlp|self_attn).(up|down|gate|q|k|v|o)_proj'
lora_modules_to_save:
  - embed_tokens
  - lm_head

# === Data Configuration ===
datasets:
  - path: gemma3.parquet
    shuffle: true
    ds_type: parquet
    type:

dataset_prepared_path: last_run_prepared

# === Hardware Optimization ===
gradient_checkpointing: true
activation_checkpointing: true

# === Wandb Tracking ===
wandb_project: g4_12b_torture

# === Checkpointing ===
saves_per_epoch: 4

# === Advanced Settings ===
output_dir: ./model-output
bf16: auto
train_on_inputs: false
group_by_length: false
logging_steps: 1
trust_remote_code: false
attn_implementation: flash_attention_2
gemma4_hybrid_attn_impl: true

plugins:
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true