Magnum "Diamond" in reference to the intense heat and pressure (generated through matrix multiplications) needed to turn the coal-esque material of dry, assistant-tuned models into creative writing gems!
This model is finetuned from a text-only conversion of mistralai/Mistral-Small-3.2-24B-Instruct-2506 as an rsLoRA adapter. It uses the same data mix as Doctor-Shotgun/L3.3-70B-Magnum-v5-SFT-Alpha, however with pre-tokenization and modifications to the custom loss masking. The goal was to re-create the model at a smaller, more consumer-friendly size. This model should perform competently with or without prepending character names, and with or without prefill.
The objective, as with the other Magnum models, is to emulate the prose style and quality of the Claude 3 Sonnet/Opus series of models on a local scale, so don't be surprised to see "Claude-isms" in its output.
This is a minor version update over Doctor-Shotgun/MS3.1-24B-Magnum-Diamond-LoRA utilizing the new official instruct model from June 2025.
This model is intended for creative writing and roleplay purposes. It may show biases similar to those observed in contemporary LLM-based roleplay, in addition to those exhibited by the Claude 3 series of models and the base model. All outputs should be considered fiction, as this model is not intended to provide factual information or advice.
There was a weird loss spike of unclear significance on one sample that was not seen using the same dataset on Mistral Small 3.1 Instruct, but the resulting model appears to be sane.
See axolotl configaxolotl version: 0.9.2
base_model: anthracite-core/Mistral-Small-3.2-24B-Instruct-2506-Text-Only
#base_model_ignore_patterns: "consolidated.safetensors"
# optionally might have model_type or tokenizer_type
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
# Automatically upload checkpoint and final model to HF
hub_model_id: NewEden/magnum-v5-sft-prototype-ms3.2-lora
hub_strategy: "all_checkpoints"
push_dataset_to_hub:
hf_use_auth_token: true
load_in_8bit: false
load_in_4bit: false
strict: false
datasets:
- path: NewEden/magnum-v5-sft-proto-mistral-v7-tekken-rev1-32k
ds_type: parquet
type:
shuffle_merged_datasets: true
dataset_prepared_path: ./magnum-24b-data
val_set_size: 0.0
output_dir: ./magnum-24b-lora-out
plugins:
- axolotl.integrations.liger.LigerPlugin
- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
liger_rope: true
liger_rms_norm: true
liger_layer_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: false
cut_cross_entropy: true
sequence_len: 32768
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true
adapter: lora
lora_model_dir:
lora_r: 128
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
peft_use_rslora: true
lora_modules_to_save:
- embed_tokens
- lm_head
wandb_project: 24b-magnum-lora
wandb_entity:
wandb_watch:
wandb_name: 24b-magnum-lora-mistral-3.2
wandb_log_model:
gradient_accumulation_steps: 16
micro_batch_size: 1
num_epochs: 2
optimizer: paged_ademamix_8bit
lr_scheduler: cosine
learning_rate: 2e-5
max_grad_norm: 1.0
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention:
warmup_steps: 40
evals_per_epoch:
eval_table_size:
eval_max_new_tokens:
saves_per_epoch: 2
debug:
deepspeed:
weight_decay: 0.01
fsdp:
fsdp_config:
special_tokens:
The following hyperparameters were used during training:
checkpoint-378/optimizer.pt 6.3 GBcheckpoint-567/optimizer.pt 6.3 GBcheckpoint-754/optimizer.pt 6.3 GBcheckpoint-189/optimizer.pt 6.3 GBadapter_model.safetensors 5.6 GBcheckpoint-189/adapter_model.safetensors 5.6 GBcheckpoint-378/adapter_model.safetensors 5.6 GBcheckpoint-567/adapter_model.safetensors 5.6 GBcheckpoint-754/adapter_model.safetensors 5.6 GBcheckpoint-189/rng_state.pth 14.6 kBcheckpoint-378/rng_state.pth 14.6 kBcheckpoint-567/rng_state.pth 14.6 kBcheckpoint-754/rng_state.pth 14.6 kBcheckpoint-189/training_args.bin 7.1 kBcheckpoint-378/training_args.bin 7.1 kBcheckpoint-567/training_args.bin 7.1 kBcheckpoint-754/training_args.bin 7.1 kBtraining_args.bin 7.1 kBcheckpoint-189/scheduler.pt 1.5 kBcheckpoint-378/scheduler.pt 1.5 kBcheckpoint-567/scheduler.pt 1.5 kBcheckpoint-754/scheduler.pt 1.5 kBHave the files? Submit a torrent.
Submit a torrentHave the complete files and permission to share them? Replace the example folder path. The script checks the files, makes a torrent and submits its magnet and checksums. Keep your torrent client seeding. Get your community key from your settings, and read the script before running it.
curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo Doctor-Shotgun/MS3.2-24B-Magnum-Diamond-LoRA ./model-foldermistralai/Mistral-Small-3.2-24B-Instruct-2506