Sign up

developerjeremylive/NAVA-etheroi

🤗 On Hugging Facetext-to-videoapache-2.072 GBother✓ HF checksums availableupdated today
Magnet

NAVA — Native Audio-Visual Alignment for Generation

State-of-the-art audio-visual synchronization with only 6.3 B parameters.

ERNIE Team · Baidu Inc. · arXiv 2026

⭐ If you find this model useful, please consider giving our GitHub repo a star! ⭐

📖 中文版 README

TL;DR

NAVA is a 6.3 B-parameter joint audio-video generator that synthesizes synchronized video and audio from a single prompt — including multi-speaker speech with reference-timbre control and image-conditioned continuations.

Instead of post-hoc-aligned dual towers or fully unified tri-modal stacks, NAVA uses an Align-then-Fuse MMDiT: a dedicated alignment space first establishes audio-video correspondence, then context (text, speaker embeddings) is fused via cross-attention. On Verse-Bench it sets new SOTA on Sync-C / Sync-D / video quality / audio WER while using 2× to 5× fewer parameters than open-source baselines.

Highlights
- 720p 1-min Fast Generation — 720p synchronized audio-video in ~1 minute via 8-GPU Ulysses sequence parallel.
- Dual-Channel Audio — stereo audio (scene + speech) jointly denoised with video, no post-hoc vocoder alignment.
- Precise Multi-Timbre Control — reference WAVs bound to ... speech spans for per-speaker voice identity.
- Language-Described Camera Control — shot composition, motion, and pacing directly from the prompt.
- Multi-Resolution — landscape / portrait / square aspect ratios from the same checkpoint.

Model Details

Quick Facts

| | |

|---|---|

| Architecture | Align-then-Fuse MMDiT (Wan2.2 backbone) |

| Parameters | 6.3 B (backbone, joint AV) |

| Modality | Joint audio + video, text-conditioned |

| Resolution | 1280×704 (recommended) · 960×960 also supported |

| Frames / FPS | 37 frames @ 24 fps ≈ 6 s · 55–61 frames ≈ 9–10 s |

| Audio | 25 latent tokens / sec, ≤ 10 s |

| Sampling | Flow matching · UniPC scheduler · 50 default steps |

| Precision | bf16 |

| Parallelism | Single-GPU or Ulysses sequence parallel (up to 8 GPUs) |

| Base model | Wan-AI/Wan2.2-TI2V-5B |

Architecture

NAVA instantiates Native Audio-Visual Alignment as an Align-then-Fuse MMDiT stack:

  • Hierarchical Alignment Layers — 10 double-stream blocks. Video and audio keep separate QKV projections and FFNs but share a joint self-attention over concatenated [video_tokens; audio_tokens], plus dedicated cross-attention to text. This builds an alignment space where AV correspondence is learned without semantic context interference.
  • Unified Fusion Layers — 20 single-stream blocks. Video and audio share QKV/FFN; a unified joint attention treats all tokens as one stream, with a single text cross-attention path. This is where context-conditioned denoising happens.
  • Backbone hyperparameters. dim=3072, ffn_dim=14336, 24 attention heads, 30 layers (10 double + 20 single), text_len=512, patch size (1, 2, 2). RMSNorm on QK; cross-attention norm; ε = 1e-6.
  • Positional encoding. 3D RoPE for video (temporal + height + width), 1D RoPE for audio, applied jointly inside the joint-attention path.
  • Timbre-in-Context Conditioning. Reference-WAV speaker embeddings (ReDimNet, 192-d) are injected through the context pathway and bound to ... speech spans, enabling per-speaker timbre control in multi-speaker scenes.
  • 3D cross-modal CFG. Independent classifier-free guidance scales for video, audio, and the cross-modal alignment direction (video_align_guidance_scale, audio_align_guidance_scale) keep AV synchronization tight at inference.

What's Different from Existing Open-Source AV Models

| Design axis | Typical baselines | NAVA |

|---|---|---|

| Stream layout | Dual-tower (post-hoc align) or fully unified tri-modal | Align-then-Fuse — alignment space first, context fused after |

| Speech control | Caption-only, no per-speaker timbre | Timbre-in-Context via reference WAVs |

| Param budget | 10 B – 32 B | 6.3 B |

Components Shipped Alongside the Backbone

| Component | Description | Size |

|---|---|---|

| WanAVModel (backbone) | MMDiT, joint AV attention | 6.3 B |

| Wan2.2 Video VAE | Causal 3D ConvNet · 16×16×4 spatial-temporal compression · 48 latent channels | 2.7 GB |

| LTX Audio VAE + Vocoder | 128 latent channels · 25 tokens/sec · built-in waveform decoder | 348 MB |

| umt5-xxl Text Encoder | T5 · 4096-d embeddings | 11 GB |

| ReDimNet | Speaker embedding · 192-d | ~50 MB |

Evaluation

Table 1 — VerseBench (general AV capability)

NAVA achieves the best AV synchronization (Sync-C / Sync-D), video quality, and audio WER, with the smallest parameter budget.

| Model | Params | Resolution | Sync-C ↑ | Sync-D ↓ | IB ↑ | Video Quality ↑ | WER ↓ | PQ ↑ | FD ↓ |

|---|---|---|---|---|---|---|---|---|---|

| Ovi 1.1 | 10 B | 720p | 7.4839 | 7.9791 | 0.199 | 0.636 | 0.102 | 5.8432 | 0.9418 |

| MOVA | A18B (32 B) | 720p | 7.2888 | 7.808 | 0.269 | 0.603 | 0.126 | 7.2331 | 0.9222 |

| Davinci | 15 B | 540p | 7.1487 | 7.8158 | 0.269 | 0.600 | 0.151 | 5.9559 | 0.9307 |

| LTX 2.3 | 19 B | 512p | 7.2476 | 7.6902 | 0.337 | 0.576 | 0.106 | 6.9459 | 0.8287 |

| NAVA (ours) | 6.3 B | 720p | 7.7914 | 7.5655 | 0.313 | 0.659 | 0.099 | 6.8609 | 0.8328 |

↑ higher is better · ↓ lower is better · bold = best · underline = 2nd best.

Table 2 — Seed-TTS-eval (speech quality)

Among joint AV models, NAVA delivers speech quality close to dedicated audio-only systems. Audio-only rows are listed for reference; they are not directly comparable.

| Category | Model | WER ↓ | Speaker Similarity ↑ |

|---|---|---|---|

| Audio-Only (reference) | CosyVoice | 4.29 | 60.9 |

| Audio-Only (reference) | Qwen2.5-Omni | 2.72 | 63.2 |

| Audio-Video Joint | DreamID-Omni | 33.44 | 34.1 |

| Audio-Video Joint | NAVA (ours) | 5.81 | 62.4 |

How to Use

TL;DR command. After §1 setup is complete:
```bash
bash scripts/inference.sh # General T2AV
bash scripts/inference_timbre.sh # I2AV + timbre control
```
Outputs land under eval_results/.

1 · Setup (once)

git clone https://github.com/ernie-research/NAVA && cd NAVA

# Python deps
pip install torch torchvision torchaudio
pip install diffusers transformers accelerate safetensors einops scipy PyYAML tqdm sentencepiece
pip install flash-attn --no-build-isolation

# All weights in one shot — main checkpoint + Wan2.2 VAE + T5 + LTX audio VAE
huggingface-cli download  --local-dir .

Expected on-disk layout

NAVA/
├── NAVA.ckpt                                                    # main checkpoint (24 GB)
├── Wan2.2-TI2V-5B/
│   ├── Wan2.2_VAE.pth                                           # 2.7 GB
│   ├── models_t5_umt5-xxl-enc-bf16.pth                          # 11 GB
│   └── google/umt5-xxl/{spiece.model, tokenizer.json}
├── params/
│   └── LTX2/
│       ├── ltx-2.3-22b-dev_audio_vae.safetensors                # 348 MB
│       └── LICENSE                                              # LTX-2 Community License
└── configs/                                                     # inference YAMLs

The LTX audio-VAE Python code is vendored under nava_src/vendor/ltx_core/ (see its NOTICE.md), so no separate clone of the LTX-Video repo is needed. ReDimNet is fetched via torch.hub on first run.

2 · One-command inference (recommended, 8 GPU SP)

The repo ships two end-to-end scripts that build a JSONL inline and launch SP=8 inference:

# General T2AV (text-only)
bash scripts/inference.sh

# I2AV + Timbre Control (first-frame image + reference voice)
bash scripts/inference_timbre.sh

Override defaults via env vars:

CKPT=/path/to/NAVA.ckpt OUT_DIR=eval_results/run1 bash scripts/inference.sh
TIMBRE_SCALE=3.0 SPK_WAV=/path/to/spk.wav    bash scripts/inference_timbre.sh

3 · Custom batches — write your own JSONL

Each line is one prompt:

{"prompt": "一位男子在海边奔跑，镜头跟随。背景是海浪声和风声。"}
{"prompt": "两人对话HelloHi there", "spk_wavs": ["spk1.wav", "spk2.wav"]}
{"prompt": "镜头跟随主体...", "image_path": "/abs/path/first_frame.png"}

| Field | Required | Description |

|---|---|---|

| prompt | yes | Text caption (also accepts legacy text field name) |

| image_path | no | Absolute path to first-frame image — auto-enables I2V for this sample |

| spk_wavs | no | List of absolute paths to speaker reference WAVs (max 2) |

Then launch:

SETUPTOOLS_USE_DISTUTILS=stdlib torchrun \
    --nnodes=1 --nproc_per_node=8 \
    --master_addr=127.0.0.1 --master_port=29507 \
    inference_nava.py \
    --config configs/baseline_t2av_demo_mmdit_no_split_ltx_control_unipc.yaml \
    --ckpt NAVA.ckpt \
    --out_dir ./outputs \
    --data_format json --data_file my_prompts.jsonl \
    --width 1280 --height 704 --frames 37 --fps 24 \
    --steps 50 --save_sample --gen_turn 1 --use_sp

Outputs land at outputs/{save_path}-{gen_turn}_av.mp4. For timbre-controlled samples, also pass --timbre_cfg --timbre_align_guidance_scale 3.0.

Mode cheatsheet

| Goal | JSONL fields | Extra flags |

|---|---|---|

| Text → AV | prompt | — |

| Image → AV | prompt + image_path | (auto-detected) |

| Timbre-controlled speech | prompt + spk_wavs | --timbre_cfg --timbre_align_guidance_scale 3.0 |

| 9-second video | any | --frames 55 |

| Single-GPU (slower) | any | omit --use_sp |

4 · Prompt rewriting (recommended for short / English inputs)

NAVA is trained on Chinese dense captions; short or English prompts benefit substantially from rewriting before inference. Three pathways are provided, all sharing the same system prompt and sampling profile (so output style stays consistent), with ... speech spans preserved verbatim.

| Pathway | Backend | Speed | Best for |

|---|---|---|---|

| vLLM batch server (pe_src/) | Qwen3-4B-Thinking-2507 served via vLLM, async HTTP | < 2 s / prompt | Offline batches |

| Local transformers, single (gradio_demo/rewrite_single.py) | Same model, in-process | 40–80 s / prompt | One-off CLI |

| Gradio "Rewrite" button | Same as above, hosted in Gradio | 40–80 s / prompt | Interactive UI |

# Batch path: start vLLM server, then rewrite a txt of prompts
bash pe_src/start_server.sh --gpu 0 --low-footprint
python pe_src/rewrite.py -i prompts.txt -o prompts_rewritten.txt

5 · Gradio Web UI

Interactive demo with click-to-rewrite (Qwen3-4B), image upload, and reference-WAV upload:

bash gradio_demo/start_gradio.sh \
    --config configs/baseline_t2av_demo_mmdit_no_split_ltx_control_unipc.yaml \
    --ckpt NAVA.ckpt \
    --rewrite_model pe_src/Qwen3-4B-Thinking-2507 \
    --port 8000 --nproc 8

Debug mode (no models, UI only)

python gradio_demo/gradio_server.py --debug --port 8000

Bias, Safety, and Misuse

NAVA can synthesize video and speech conditioned on a reference image (image_path) and reference voice (spk_wavs). Using it to depict real persons without consent — including face-likeness or voice-likeness reproduction — is prohibited by the license and may also be illegal in your jurisdiction. We recommend:

1. Only use consent-approved reference media.

2. Label generated content as synthetic.

3. Apply provenance / watermarking before redistribution.

Citation

@article{nava2026,
  title   = {NAVA: Native Audio-Visual Alignment for Joint Audio-Video Generation},
  author  = {ERNIE Team},
  journal = {arXiv preprint},
  year    = {2026},
}

Acknowledgements

NAVA builds on excellent upstream work: Wan2.2-TI2V-5B (video backbone & VAE), LTX 2.3 (audio VAE + built-in vocoder), umt5-xxl (text encoder), and ReDimNet (speaker embedding). We also thank the open-source AV-generation community — Ovi, MOVA, Davinci, LTX — for releasing strong baselines that made fair benchmarking possible.

License & Contact

Released under Apache-2.0. For research / commercial inquiries, contact the ERNIE team at Baidu Inc.