Paper: arXiv:2611.17831
Code and technical report: github.com/kkt94/Llama-V-JEPA-LM
Live demo: Llama-V-JEPA-LM-Llama3.1-8B in your browser (ZeroGPU)
Base language model: Qwen3-1.7B (Apache-2.0). The name begins with "Llama" because the training data include the FGQA split of PLM-Video-Human, an output of Llama 3.2 whose license requires models trained on it to carry the "Llama" prefix.
Llama-V-JEPA-LM-Qwen3-1.7B is a video-language model from the Llama-V-JEPA-LM family. It joins a frozen V-JEPA 2.1 world-model video encoder (ViT-L) to the Qwen3-1.7B language model through a spectral bridge that turns every 32-frame clip into 1,024 visual tokens. It is trained with counterfactual supervision: pixel transformations of public videos produce answer-balanced variants whose answers are fixed by construction and differ only in how the scene changes, so the language model learns to read the dynamics (direction, speed, order, state) that the world-model encoder carries. Trained on 4.60M public samples, Llama-V-JEPA-LM-Qwen3-1.7B reaches a 66.02 average on 12 scores across 11 video benchmarks, ahead of every open model of its size class (best open 2B-class model: Qwen3-VL-2B, 52.45).
| Model type | Video-language model (video + text in, text out) |
| Language model | Qwen3-1.7B, fully fine-tuned (1.72 B parameters) |
| Vision encoder | V-JEPA 2.1 ViT-L (distilled from ViT-G), frozen (0.30 B parameters) |
| Bridge | spectral bridge: fixed PCA rotation and rescaling to 1,024 dimensions + trainable MLP (6.6 M parameters) |
| Total parameters | 2.03 B |
| Visual tokens | 1,024 per clip (16 tubelets × 8 × 8 cells), independent of input resolution |
| Frames | 32 per clip, sampled uniformly, 384 × 384 |
| Context | 2,560 tokens in training (video tokens + prompt + answer) |
| Precision | bf16 (bridge projection statistics in fp32) |
| Training samples | 4.60M (drawn at fixed per-source weights from a mixture of 175 public sources) |
| Language | English |
| Licence | Apache-2.0; V-JEPA 2 encoder: MIT; FGQA training data: Llama 3.2 Community License (see License) |
| Component | Details | Parameters |
|---|---|---|
| Video encoder (frozen) | V-JEPA 2.1 ViT-L/16 distilled from ViT-G (vjepa2_1_vitl_dist_vitG_384.pt, EMA encoder), 384 x 384, tubelet 2, RoPE; its last output level (block 23) |
0.30 B |
| Spectral bridge | LayerNorm of the last level, 3 x 3 patch pooling to 8 x 8 cells per tubelet, projection onto the top 1,024 principal directions of the frozen V-JEPA features with one global scale (fixed statistics, fp32), LayerNorm + MLP 1024 -> 2048 -> 2048, plus learned position, time and cell-geometry codes | 6.6 M (+ 1.05 M fp32 statistics) |
| Language model (fully fine-tuned) | Qwen3-1.7B architecture (28 layers, hidden 2048), tied input/output embeddings | 1.72 B |
Weights are bf16 safetensors (1 file, about 4.1 GB). The only exception is the bridge's projection statistics (bridge.spectral.{mean,basis,scale}), which stay in fp32, also after .to(torch.bfloat16).
1. Install (Python 3.10+, a CUDA build of PyTorch that matches your driver):
pip install torch
pip install "transformers>=5" safetensors tokenizers numpy jinja2 eva-decord huggingface_hub
2. Run on a GPU with 12 GB or more (peak memory for one clip in our test: 4.2 GiB):
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained(
"kkt94/Llama-V-JEPA-LM-Qwen3-1.7B",
trust_remote_code=True, dtype=torch.bfloat16,
).to("cuda").eval()
# free-form answer (greedy decoding)
print(model.chat("my_video.mp4", "Describe what happens in the video."))
# multiple choice -> (index of the chosen option, one score per option)
idx, scores = model.choose("my_video.mp4", "Which way does the ball roll?", ["to the left", "to the right"])
print(idx, scores)
# only part of the video (seconds)
print(model.chat("my_video.mp4", "What does the person pick up?", start_sec=5.0, end_sec=12.0))
# encode once, ask several questions
frames, times = model.load_video("my_video.mp4")
ev = model.encode_video(frames, times) # [1, 1024, 2048] video tokens
print(model.choose(None, "Is the door open at the end?", ["yes", "no"], evidence=ev))
print(model.chat(None, "What happens next?", evidence=ev))
hf download kkt94/Llama-V-JEPA-LM-Qwen3-1.7B --local-dir ./Llama-V-JEPA-LM-Qwen3-1.7B downloads the files (about 4.1 GB) if you prefer a local folder; from_pretrained accepts that folder too.
choose scores the options and does not generate. The prompt is "{question}\nA. {option 0}\nB. {option 1}\n...\nAnswer with the option's letter from the given choices directly."; the prediction is the argmax of the option-letter logits at the first answer position (up to 10 options, A-J). This is the protocol of the evaluation below.chat answers a single-turn question with greedy decoding (max_new_tokens, default 256).chat_template.jinja, set on model.tokenizer automatically): Qwen3 <|im_start|> turns in Qwen3's non-thinking format (every assistant turn starts with an empty <think></think> block), no system prompt. The video tokens come first in the single user turn, followed by the question. The API takes one video and one question per call.t0 + (i + 0.5) * (t1 - t0) / 32 s, nearest source frame); the clip is the whole video or [start_sec, end_sec]. Short side resized to 384 px (bicubic, antialiased), centre crop 384 x 384, ImageNet normalisation; the real frame timestamps are encoded into the video tokens. For long videos, start_sec / end_sec select the segment of interest.decord reads (mp4, webm, mkv, avi, ...).rope_inv_freq_bf16 (config, default true) rounds the LLM's rotary frequencies to bf16 as in the evaluation these scores come from.| File | Contents |
|---|---|
model-*.safetensors, model.safetensors.index.json |
weights (vision encoder, bridge, language model) |
config.json |
JepaLlmConfig (model_type: jepa_llm) with the language model's config as text_config |
configuration_jepa_llm.py |
the config class |
modeling_jepa_llm.py |
JepaLlmModel: encoder, bridge, language model and the chat / choose API |
processing_jepa_llm.py |
video decoding, frame sampling, preprocessing, prompt building, chat templates |
vjepa2_1_encoder.py |
inference-only copy of the V-JEPA 2.1 encoder (MIT) |
tokenizer files, chat_template.jinja |
the language model's tokenizer and the template the model was trained with |
LICENSE_QWEN3, LICENSE_VJEPA2, LICENSE_LLAMA3_2, NOTICE_LLAMA3_2, USE_POLICY_LLAMA3_2.md |
licences (*_LLAMA3_2: Llama 3.2 licence, notice and use policy for the PLM-Video-Human FGQA training data) |
The code needs no external repository (trust_remote_code=True). Tested with Python 3.12, torch 2.13, transformers 5.10.4 (see requirements.txt).
Twelve scores on eleven video benchmarks for Llama-V-JEPA-LM-Qwen3-1.7B and every open model of the 2B class. All models are evaluated by the same harness with one protocol: 32 frames, the same items and prompts, option-letter scoring. Bold: best in the table.
| Model | MVB | PT | TVB | TC | TOM | MoB | NQA | MVP | TB | SSv2 | VG-T | VG-V | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-V-JEPA-LM-Qwen3-1.7B | 69.5 | 75.3 | 70.7 | 73.5 | 50.9 | 54.3 | 74.0 | 54.6 | 63.9 | 91.6 | 63.2 | 50.8 | 66.02 |
| Qwen3-VL-2B | 61.0 | 65.1 | 49.8 | 67.4 | 30.3 | 57.5 | 77.1 | 32.3 | 65.9 | 59.2 | 36.2 | 27.6 | 52.45 |
| InternVL3.5-2B | 66.8 | 64.2 | 50.9 | 65.7 | 31.0 | 56.2 | 76.1 | 28.8 | 71.3 | 57.0 | 40.0 | 21.0 | 52.42 |
| PLM-1B | 68.0 | 69.8 | 48.2 | 60.5 | 24.5 | 51.7 | 79.8 | 30.9 | 58.1 | 61.7 | 33.2 | 29.6 | 51.33 |
| Apollo-1.5B | 61.1 | 63.9 | 47.9 | 60.3 | 27.8 | 51.1 | 78.8 | 26.0 | 55.8 | 56.0 | 38.8 | 22.4 | 49.16 |
| VideoLLaMA3-2B | 57.9 | 62.9 | 44.0 | 64.1 | 24.0 | 56.3 | 75.4 | 23.1 | 64.1 | 50.0 | 38.0 | 25.2 | 48.75 |
| Gemma 4 E2B | 39.8 | 46.3 | 36.4 | 51.5 | 18.7 | 41.1 | 51.4 | 8.1 | 61.4 | 37.0 | 24.0 | 22.4 | 36.51 |
MVB: MVBench, PT: PerceptionTest, TVB: TVBench, TC: TempCompass, TOM: TOMATO, MoB: MotionBench, NQA: NExT-QA, MVP: Minimal Video Pairs (pair accuracy), TB: TemporalBench (binary), SSv2: Something-Something v2 (multiple choice), VG-T / VG-V: Vinoground text / video (pair accuracy), Avg: mean of the 12 scores.
Held-out generalization. Six construction sets built from videos and settings never used in training (unseen playback-speed ratios, a new source corpus for speed and for direction, unseen mirror × reversal and acceleration × reversal combinations, camera motion from an unseen simulator; chance 33.3): 95.0 average for Llama-V-JEPA-LM-Qwen3-1.7B, vs. 43.8 for Qwen3-VL-2B.
Inference cost per 32-frame clip on one H100 (batch 1, bf16, video decoding excluded; times: median of 10 runs after 3 warm-ups; 480p clip unless noted):
| Model | Visual tokens | Encoder (TFLOPs) | LLM prefill (TFLOPs) | LLM prefill (ms) | First token, 480p (ms) | Decode (ms / token) | First token, 720p (ms) |
|---|---|---|---|---|---|---|---|
| Llama-V-JEPA-LM-Qwen3-1.7B | 1,024 | 13.9 | 3.3 | 22 | 83 | 20.0 | 82 |
| Qwen3-VL-2B | 1,920 / 11,520 | 5.0 | 7.0 | 31 | 66 | 18.2 | 308 |
| InternVL3.5-2B | 8,192 | 23.2 | 40.5 | 104 | 159 | 18.4 | 159 |
| PLM-1B | 8,192 | 22.2 | 25.2 | 57 | 146 | 9.3 | 146 |
First token = vision encoder + connector + LLM prefill. Decode: per generated token with KV cache. --: not timed. The visual-token count and cost of Llama-V-JEPA-LM models do not depend on the input resolution. The encoder time is a property of the encoder: V-JEPA 2.1 attends jointly over all 32 frames (9,216 patches), which is what carries the dynamics, while the open models use frame-wise image encoders (SigLIP-2-based in Qwen3-VL, InternViT, PE) that attend within one frame or frame pair at a time; it is identical for every Llama-V-JEPA-LM model on the same encoder.
Data. 4.60M samples drawn at fixed per-source weights from 175 public sources: open video caption and QA collections (LLaVA-Video, ShareGPTVideo, PerceptionLM data, Charades-Ego, EPIC-Kitchens, HoloAssist, SSv2, Kinetics-700, SA-V, ActivityNet and others) plus 33 counterfactual sources (509,193 rows, about 6% of the drawn samples). The counterfactual rows come from 14 constructions in 6 families. Each takes a clip of a public dataset and a transformation; the answer follows from the transformation, so no new human annotation and no language model are involved. Within each source every scene appears with each answer equally often in expectation.
| Family | Constructions | Rows |
|---|---|---|
| Sign flip | clip reversed or mirrored, camera move rendered or reversed, two clips each optionally reversed | 117,066 |
| Order | two events joined in sequence or stacked; which segment matches a two-action caption | 85,579 |
| Position | several clips joined, one holds the queried action: beginning / middle / end | 48,386 |
| Magnitude | relative speed, acceleration / deceleration, slow motion vs. time-lapse | 141,084 |
| Count | one action repeated k times; number of distinct activities | 35,614 |
| Stop/cut | clip cut before the action completes; physically impossible simulator edits | 81,464 |
Recipe. The V-JEPA 2.1 encoder stays frozen. After a bridge-only warm-up (0.6–0.8% of the steps), the bridge and the full language model are trained together in one continuous stage: 27,382 steps at global batch 168, learning rate 1.05e-5 for the language model and 2e-4 for the bridge, cosine decay, weight decay 0.01 / 0.05, maximum sequence length 2,560 tokens.
Hardware. 14 NVIDIA H100 GPUs (12 clips per GPU), 16.4 hours, 229 H100-hours in total; training compute 121 EFLOPs (including the frozen encoder's forward pass).
Fine-tuned on the training splits of public video-language datasets; no new human annotation.
| Dataset | License | Note |
|---|---|---|
| Something-Something v2 | Qualcomm research-use license | non-commercial |
| LLaVA-Video-178K | Apache-2.0 (card: academic research) | GPT-4o-generated |
| ShareGPTVideo (LLaVA-Hound) | Apache-2.0 | GPT-generated |
| HoloAssist | CDLA-Permissive-2.0 | – |
| SA-V | CC BY 4.0 | – |
| PLM-Video-Human (RCap, RDCap, RTLoc) | CC BY 4.0 | – |
| PLM-Video-Human FGQA | Llama 3.2 Community License | Llama 3.2 output |
| Charades / Charades-Ego | AI2 non-commercial license | non-commercial |
| EPIC-KITCHENS-100 | CC BY-NC 4.0 | non-commercial |
| EgoDex | CC BY-NC-ND 4.0 | non-commercial |
| Kinetics-700 | CC BY 4.0 (annotations) | YouTube videos |
| NVIDIA PhysicalAI Physical-Interaction-Scenes | OpenMDW-1.1 | – |
| RoboVQA | CC BY 4.0 | – |
| Language-Table | Apache-2.0 | – |
| CLEVRER | see project page | – |
| TimeIT | CC BY 4.0 (instructions) | source videos under their own terms |
| PerceptionTest (train) | CC BY 4.0 | – |
| STAR (train) | Apache-2.0 | Charades videos |
| Oops! | CC BY-NC-SA 4.0 | non-commercial |
| HT-Step | CC BY-NC 4.0 | non-commercial |
| Others (ActivityNet, YouCook2, LSMDC, Ego-QA, Video-Holmes, VidLN, Countix) | see each dataset | research use |
| UltraChat-200k prompts | MIT | answers generated by the base LLM |
| Reasoning / answer rewrites | generated by Qwen3-8B / Qwen3-VL-8B (Apache-2.0) | – |
Each dataset remains under its own terms; this release grants no rights in them.
Several training datasets restrict commercial use. We release the weights for research and non-commercial use; anyone considering commercial use is responsible for checking the terms of the training data.
start_sec / end_sec select the segment to ask about.This model combines components under the following licences:
LICENSE_QWEN3): all weights were fine-tuned on video-language data, and video tokens from the bridge are inserted into its input embeddings. The tokenizer files are unmodified copies from Qwen3-1.7B.vjepa2_1_vitl_dist_vitG_384.pt released by Meta with facebookresearch/vjepa2 (MIT, LICENSE_VJEPA2), unchanged (frozen). vjepa2_1_encoder.py is adapted from that repository under the MIT License.LICENSE_LLAMA3_2) and the Llama 3.2 Acceptable Use Policy (USE_POLICY_LLAMA3_2.md); as that license requires for models trained on it, the model name begins with "Llama". Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. (NOTICE_LLAMA3_2)This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP, AI Computing Support Project for R&D) grant funded by the Korea government (MSIT) (No. RS-2026-25505492, High-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale).
@misc{kim2026vjepalm,
title = {V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations},
author = {Kim, Keuntae and Kim, Dongjun and Choi, Yong Suk},
year = {2026},
eprint = {2611.17831},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2611.17831}
}
Please also cite V-JEPA 2 / V-JEPA 2.1 and Qwen3.
All sizes are in the Llama-V-JEPA-LM collection.
| Model | Language model | Encoder | Avg (12) |
|---|---|---|---|
| Llama-V-JEPA-LM-Llama3.2-1B | Llama-3.2-1B-Instruct | V-JEPA 2.1 ViT-L | 62.21 |
| Llama-V-JEPA-LM-Qwen3-1.7B (this model) | Qwen3-1.7B | V-JEPA 2.1 ViT-L | 66.02 |
| Llama-V-JEPA-LM-Llama3.2-3B | Llama-3.2-3B-Instruct | V-JEPA 2.1 ViT-g | 68.97 |
| Llama-V-JEPA-LM-Qwen3-4B | Qwen3-4B-Instruct-2507 | V-JEPA 2.1 ViT-g | 72.14 |
| Llama-V-JEPA-LM-Llama3.1-8B | Llama-3.1-8B-Instruct | V-JEPA 2.1 ViT-g | 70.51 |
| Llama-V-JEPA-LM-Qwen3-8B | Qwen3-8B | V-JEPA 2.1 ViT-g | 70.11 |
| Llama-V-JEPA-LM-Qwen3-30B-A3B | Qwen3-30B-A3B | V-JEPA 2.1 ViT-G | 69.49 |
Keuntae Kim, Hanyang University, Artificial Intelligence Lab
Qwen/Qwen3-1.7B