JoyFox LiveTalk-DH 1.3B
Any Image · Any Voice · Bring Every Persona to Life
JoyFox LiveTalk-DH 1.3B is a custom-trained model for high-quality,
audio-driven human video generation. It was retrained from the official
base model, with a focus on high-resolution portrait detail, identity
consistency, lip synchronization, and long-sequence stability.
Provide one portrait image and one driving audio track to generate a human
video that preserves the original composition, identity, and background.
Inference uses the complete reference image without center cropping or
adding white padding.
Visual Comparisons
Each comparison uses the same reference image, driving audio, and inference
settings. The input retains its original aspect ratio and is scaled only to a
VAE-compatible aligned resolution. No cropping, padding, or recomposition is
applied. The left side shows SoulX-FlashHead-1_3B Lite; the right side shows
JoyFox-LiveTalk-DH.
Natural Portrait
Reference Image
Side-by-Side Comparison
Studio Portrait
Reference Image
Side-by-Side Comparison
Model Highlights
| Capability | Design | Benefit |
| :--- | :--- | :--- |
| 🎭 Identity Preservation | Spatial VAE reference conditioning | Preserves facial structure, silhouette, clothing colors, and background layout |
| 🖼️ Original Composition | Multi-aspect-ratio buckets and automatic size matching | Uses the full reference image without cropping or white padding |
| 🔊 Audio Driven | Multi-layer wav2vec2 features and audio cross-attention | Improves synchronization between lip motion, speech rhythm, and facial movement |
| 🎬 Continuous Generation | 97-frame training and streaming segment continuity | Improves long-sequence motion and transitions between generated segments |
| ✨ High-Definition Detail | Retrained at a 960-pixel baseline | Enhances portrait texture and fine detail at high resolution |
| ⚡ Efficient Deployment | 1.3B Transformer with BF16 weights | Balances generation quality with inference cost |
| 🔓 Simplified Dependencies | Official LTX Video VAE | Requires neither a text encoder nor a CLIP image encoder |
JoyFox LiveTalk-DH
JoyFox LiveTalk-DH is the high-resolution, multi-aspect-ratio training and
inference method used by this release. It does not rely on an additional text
encoder or CLIP image encoder. Instead, the reference image, audio, and video
timeline are jointly modeled within one generation network.
Multi-Aspect-Ratio Bucketing
Training samples are not stretched into a fixed 512×512 square. The system
selects the closest resolution bucket from the source video aspect ratio and
aligns the spatial dimensions to the compression requirements of the LTX VAE.
The 960-pixel baseline provides more effective pixels while allowing landscape,
portrait, and square content to retain more natural geometry.
Composition Preservation
At inference time, the original image dimensions are read and the output aspect
ratio is selected automatically. For non-standard inputs, direct mode
prioritizes the source aspect ratio and selects the nearest dimensions aligned
to a multiple of 32. The full image is scaled proportionally without center
cropping, padding, or synthetic background extension.
Spatial VAE Reference Conditioning
The official LTX Video VAE encodes the reference image into a 128-channel video
latent space. The reference latent acts as first-frame motion context and is
expanded into video conditioning that enters the Transformer together with the
noise latents. Compared with a single image embedding, this spatial condition
retains facial structure, subject contours, colors, and background layout.
Audio Cross-Attention
wav2vec2 extracts multi-layer temporal features from the driving speech. A
local audio window is constructed for each video frame and converted into
context tokens by the audio projection network. Cross-attention in the
Transformer layers aligns lip motion with the current, preceding, and following
speech segments.
High-Resolution Long-Sequence Training
The model was retrained with a 960-pixel baseline and 97-frame clips. Training
jointly optimizes spatial detail, inter-frame motion, and audio synchronization,
enabling more stable long sequences while preserving identity. The model is
stored in BF16 to reduce inference memory and storage requirements while
maintaining generation quality.
Streaming Segment Generation
Long-audio inference uses segmented generation with historical motion-frame
continuity. Each new segment reuses preceding motion context while a sliding
window continuously updates the audio features. This supports videos longer
than a single training clip and reduces motion discontinuities at segment
boundaries.
Scope and Limitations
The current training distribution is dominated by close-up portraits, so
close-up and bust compositions are the most reliable. Automatic aspect-ratio
selection and composition preservation reduce geometric distortion for
non-square inputs, but results for half-body, full-body, multi-person, and
complex-background scenes still depend on the coverage of those cases in the
training data.
Inference Example
The following standalone example initializes the model components and generates
an audio-driven video:
import math
from pathlib import Path
import torch
from diffusers import AutoencoderKLLTXVideo, FlowMatchEulerDiscreteScheduler
from huggingface_hub import snapshot_download
from PIL import Image
from videox_fun.models import FlashHeadAudioEncoder, FlashHeadTransformer3DModel
from videox_fun.pipeline import FlashHeadPipeline
from videox_fun.utils.utils import (
get_image_latent,
merge_video_audio,
save_videos_grid,
)
# Model repository, inputs, and output
model_dir = Path(
snapshot_download(repo_id="joyfox/JoyFox-LiveTalk-DH-1.3B")
)
reference_image = model_dir / "assets/reference-natural-portrait.jpg"
driving_audio = model_dir / "assets/driving-audio.wav"
output_video = Path("joyfox_livetalk_dh.mp4")
# wav2vec2 is a separate official audio encoder downloaded on first use.
audio_encoder_dir = snapshot_download(
repo_id="facebook/wav2vec2-base-960h",
allow_patterns=[
"config.json",
"preprocessor_config.json",
"model.safetensors",
],
)
# Find the nearest 32-aligned size while preserving the source aspect ratio.
with Image.open(reference_image) as image:
source_width, source_height = image.size
source_ratio = source_width / source_height
target_pixels = 960**2
ideal_width = (target_pixels * source_ratio) ** 0.5
ideal_height = (target_pixels / source_ratio) ** 0.5
width_center = max(1, round(ideal_width / 32))
height_center = max(1, round(ideal_height / 32))
candidates = []
for width_step in range(max(1, width_center - 4), width_center + 5):
for height_step in range(max(1, height_center - 4), height_center + 5):
width = width_step * 32
height = height_step * 32
aspect_error = abs(math.log((width / height) / source_ratio))
area_error = abs(math.log((width * height) / target_pixels))
candidates.append((aspect_error + 0.02 * area_error, height, width))
_, height, width = min(candidates)
dtype = torch.bfloat16
device = "cuda"
transformer = FlashHeadTransformer3DModel.from_pretrained(
model_dir / "Model_Lite",
transformer_additional_kwargs={
"transformer_subpath": "./",
"dict_mapping": {
"in_dim": "in_channels",
"dim": "hidden_size",
},
},
low_cpu_mem_usage=True,
torch_dtype=dtype,
)
vae = AutoencoderKLLTXVideo.from_pretrained(
model_dir / "VAE_LTX",
low_cpu_mem_usage=True,
).to(dtype)
audio_encoder = FlashHeadAudioEncoder(str(audio_encoder_dir), "cpu")
scheduler = FlowMatchEulerDiscreteScheduler(
num_train_timesteps=1000,
shift=5.0,
use_dynamic_shifting=False,
base_shift=0.5,
max_shift=1.15,
base_image_seq_len=256,
max_image_seq_len=4096,
)
pipeline = FlashHeadPipeline(
transformer=transformer,
vae=vae,
scheduler=scheduler,
audio_encoder=audio_encoder,
).to(device)
ref_image = get_image_latent(
str(reference_image),
sample_size=[height, width],
resize_mode="direct",
)
generator = torch.Generator(device=device).manual_seed(42)
with torch.inference_mode():
video = pipeline(
ref_image=ref_image,
audio_path=str(driving_audio),
height=height,
width=width,
segment_frame_length=33,
num_inference_steps=4,
audio_guide_scale=1.0,
audio_encode_mode="stream",
shift=5.0,
fps=25,
max_frames_num=500,
stochastic_sampling=True,
generator=generator,
).videos
output_video.parent.mkdir(parents=True, exist_ok=True)
save_videos_grid(video, str(output_video), fps=25)
merge_video_audio(
video_path=str(output_video),
audio_path=str(driving_audio),
)direct mode uses the complete reference image. Together with automatic size
selection, it preserves the source aspect ratio while aligning the output
dimensions to multiples of 32. It does not introduce cropping, white borders,
or blurred background fill.
Source and License
This model was retrained from the official Lite release of
Soul-AILab/SoulX-FlashHead-1_3B
and uses the Apache-2.0 license. Use and redistribution must also comply with
the original model license and the licenses governing the training data.