中文Sign up

DiffSynth-Studio/MiniMax-H3-Text-Embeddings

🤗 Hugging Face sourceapache-2.047 MBsafetensors✓ 25 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo DiffSynth-Studio/MiniMax-H3-Text-Embeddings ./model-folder
Needs a seeder →

MiniMax-H3 Templates: Text Embeddings

This repository provides a collection of video effect templates for the MiniMax-H3 model. These templates encapsulate specific model capabilities as text embeddings, implemented via Diffusion Templates.

Showcase

Effects Gallery

art_is_explosion storm_magic
dark_magic kiss_camera
bullet_time truman_show
fire_breath blooming_flowers
four_seasons spiral_ascent

Effect Combinations

Combining effects: art_is_explosion + storm_magic

How It Works

Each template consists of a single tensor. This tensor can either replace or be combined with the output of the Text Encoder, a mechanism very similar to Textual Inversion.

While MiniMax-H3 boasts powerful base capabilities, its massive parameter count makes LoRA training challenging. Text Embeddings offer a lightweight alternative to LoRA with the following advantages:

  • Modular Capabilities: Text Embeddings serve as carriers of model functionality. Similar to agent skills, they enable specific capabilities to be saved, reused, and distributed as atomic units through model platforms.
  • Flexible Initialization: Embeddings can be initialized from text prompts, images, or videos. Simply pass the input data through the MiniMax-H3 Text Encoder to quickly generate a usable Text Embedding.
  • Trainable: Like LoRA, Text Embeddings support end-to-end training on video datasets, enabling targeted enhancement of specific generative abilities.
  • Composable: Multiple templates can be combined to jointly influence generation, allowing you to create complex and visually striking effects.

Inference and Training

Installation

Install DiffSynth-Studio:

git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e ".[all]"

Initialize a Text Embedding via Text Encoder

The following code has low VRAM requirements and can run with as little as 6GB of VRAM:

import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from safetensors.torch import save_file

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[ModelConfig(
        model_id="MiniMax/MiniMax-H3",
        origin_file_pattern="FL2VA/text_encoder/model*.safetensors",
        offload_dtype="disk",
        offload_device="disk",
        onload_dtype="disk",
        onload_device="disk",
        preparing_dtype=torch.bfloat16,
        preparing_device="cuda",
        computation_dtype=torch.bfloat16,
        computation_device="cuda",
    )],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=0,
)
prompt = "xxx"
text_embedding = pipe.export_text_embedding(prompt)
save_file({"weight": text_embedding}, "model.safetensors")

Inference with Text Embeddings via Diffusion Templates

You can load text embeddings through Diffusion Templates for inference. Optionally, load an acceleration LoRA to speed up inference (note: the text encoder is not required during this stage):

import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.diffusion.template import TemplatePipeline
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from modelscope import snapshot_download
from PIL import Image

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
template = TemplatePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[ModelConfig(
        model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/art_is_explosion/",
    )],
)
snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = template(
    pipe,
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
    template_inputs=[{}],
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)
Inference with Multiple Text Embeddings
import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.diffusion.template import TemplatePipeline
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from modelscope import snapshot_download
from PIL import Image

```python
vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
template = TemplatePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/art_is_explosion/"),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings", origin_file_pattern="models/storm_magic/"),
    ],
)
snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = template(
    pipe,
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
    template_inputs=[{"model_id": 0}, {"model_id": 1}],
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)
Inference Using the Native Pipeline
import torch
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.utils.data.audio_video import write_video_audio
from diffsynth.core.data.operators import ImageCropAndResize
from diffsynth import load_state_dict
from modelscope import snapshot_download
from PIL import Image

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
)
pipe.load_lora(
    pipe.dit,
    ModelConfig(
        model_id="lightx2v/Minimax-h3-Turbo",
        origin_file_pattern="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors",
    ),
)
text_embedding_config = ModelConfig(
    model_id="DiffSynth-Studio/MiniMax-H3-Text-Embeddings",
    origin_file_pattern="models/art_is_explosion/model.safetensors",
)
text_embedding_config.download_if_necessary()
text_embedding = load_state_dict(text_embedding_config.path)["weight"]

snapshot_download("DiffSynth-Studio/MiniMax-H3-Text-Embeddings", allow_file_pattern="assets/image_1.jpg", local_dir="data")
first_frame = ImageCropAndResize(height=1344, width=768)(Image.open("data/assets/image_1.jpg"))
video, audio = pipe(
    height=1344, width=768, num_frames=56,
    num_inference_steps=4, seed=0, flow_shift=6,
    keyframes=[first_frame], keyframe_indices=[0],
    text_embedding=text_embedding,
)
write_video_audio(
    video=video, audio=audio,
    output_path="output.mp4", fps=24, audio_sample_rate=32000,
)

Training Text Embeddings

modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "minimax_h3/MiniMax-H3-Text-Embeddings/*" --local_dir ./data/diffsynth_example_dataset

accelerate launch examples/minimax_h3/model_training/train.py \
  --dataset_base_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Text-Embeddings \
  --dataset_metadata_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Text-Embeddings/metadata.json \
  --data_file_keys "video,input_audio" \
  --extra_inputs "input_audio,input_image,template_inputs" \
  --height 832 \
  --width 480 \
  --num_frames 124 \
  --dataset_repeat 100 \
  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:video_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:audio_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-fl2va-pruned-nf4.safetensors" \
  --template_model_id_or_path "DiffSynth-Studio/MiniMax-H3-Text-Embeddings:models/art_is_explosion/" \
  --learning_rate 1e-4 \
  --num_epochs 2 \
  --remove_prefix_in_ckpt "pipe.template_model." \
  --output_path "./models/train/MiniMax-H3-Text-Embeddings-full" \
  --trainable_models "template_model" \
  --use_gradient_checkpointing