Llama-V-JEPA-LM-Qwen3-1.7B
Paper: arXiv:2611.17831
Code and technical report: github.com/kkt94/Llama-V-JEPA-LM
Live demo: Llama-V-JEPA-LM-Llama3.1-8B in your browser (ZeroGPU)
Base language model: Qwen3-1.7B (Apache-2.0). The name begins with "Llama" because the training data include the FGQA split of PLM-Video-Human, an output of Llama 3.2 whose license requires models trained on it to carry the "Llama" prefix.
Llama-V-JEPA-LM-Qwen3-1.7B is a video-language model from the Llama-V-JEPA-LM family. It joins a frozen V-JEPA 2.1 world-model video encoder (ViT-L) to the Qwen3-1.7B language model through a spectral bridge that turns every 32-frame clip into 1,024 visual tokens. It is trained with counterfactual supervision: pixel transformations of public videos produce answer-balanced variants whose answers are fixed by construction and differ only in how the scene changes, so the language model learns to read the dynamics (direction, speed, order, state) that the world-model encoder carries. Trained on 4.60M public samples, Llama-V-JEPA-LM-Qwen3-1.7B reaches a 66.02 average on 12 scores across 11 video benchmarks, ahead of every open model of its size class (best open 2B-class model: Qwen3-VL-2B, 52.45).
Highlights
- Ahead of every open model of its size class. 66.02 on the 12-score average vs. 52.45 for Qwen3-VL-2B; best of all open 2B-class models on 9 of 12 scores (SSv2 91.6 vs. 61.7, Vinoground text 63.2 vs. 40.0, MVP 54.6 vs. 32.3, Vinoground video 50.8 vs. 29.6).
- Reads motion and temporal change. The counterfactual data teach direction, order, position, speed, counting and completion from public videos, with answers set by the transformation and no new human annotation.
- Generalizes beyond its constructions. 95.0 average on six held-out sets (unseen speed ratios, an unseen source corpus, unseen combinations, an unseen simulator; chance 33.3), vs. 43.8 for Qwen3-VL-2B.
- Compact and fast. 1,024 visual tokens per clip at any input resolution, so the language model prefills a clip with 3.3 TFLOPs in 22 ms on one H100, vs. 7.0–40.5 TFLOPs and 31–104 ms for the open models of its class; 83 ms to the first token in total.
Model details
| Model type | Video-language model (video + text in, text out) |
| Language model | Qwen3-1.7B, fully fine-tuned (1.72 B parameters) |
| Vision encoder | V-JEPA 2.1 ViT-L (distilled from ViT-G), frozen (0.30 B parameters) |
| Bridge | spectral bridge: fixed PCA rotation and rescaling to 1,024 dimensions + trainable MLP (6.6 M parameters) |
| Total parameters | 2.03 B |
| Visual tokens | 1,024 per clip (16 tubelets × 8 × 8 cells), independent of input resolution |
| Frames | 32 per clip, sampled uniformly, 384 × 384 |
| Context | 2,560 tokens in training (video tokens + prompt + answer) |
| Precision | bf16 (bridge projection statistics in fp32) |
| Training samples | 4.60M (drawn at fixed per-source weights from a mixture of 175 public sources) |
| Language | English |
| Licence | Apache-2.0; V-JEPA 2 encoder: MIT; FGQA training data: Llama 3.2 Community License (see License) |
| Component | Details | Parameters |
|---|---|---|
| Video encoder (frozen) | V-JEPA 2.1 ViT-L/16 distilled from ViT-G (vjepa2_1_vitl_dist_vitG_384.pt, EMA encoder), 384 x 384, tubelet 2, RoPE; its last output level (block 23) |
0.30 B |
| Spectral bridge | LayerNorm of the last level, 3 x 3 patch pooling to 8 x 8 cells per tubelet, projection onto the top 1,024 principal directions of the frozen V-JEPA features with one global scale (fixed statistics, fp32), LayerNorm + MLP 1024 -> 2048 -> 2048, plus learned position, time and cell-geometry codes | 6.6 M (+ 1.05 M fp32 statistics) |
| Language model (fully fine-tuned) | Qwen3-1.7B architecture (28 layers, hidden 2048), tied input/output embeddings | 1.72 B |
Weights are bf16 safetensors (1 file, about 4.1 GB). The only exception is the bridge's projection statistics (bridge.spectral.{mean,basis,scale}), which stay in fp32, also after .to(torch.bfloat16).
Quick start
1. Install (Python 3.10+, a CUDA build of PyTorch that matches your driver):
pip install torch
pip install "transformers>=5" safetensors tokenizers numpy jinja2 eva-decord huggingface_hub
2. Run on a GPU with 12 GB or more (peak memory for one clip in our test: 4.2 GiB):
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained(
"kkt94/Llama-V-JEPA-LM-Qwen3-1.7B",
trust_remote_code=True, dtype=torch.bfloat16,
).to("cuda").eval()
# free-form answer (greedy decoding)
print(model.chat("my_video.mp4", "Describe what happens in the video."))
# multiple choice -> (index of the chosen option, one score per option)
idx, scores = model.choose("my_video.mp4", "Which way does the ball roll?", ["to the left", "to the right"])
print(idx, scores)
# only part of the video (seconds)
print(model.chat("my_video.mp4", "What does the person pick up?", start_sec=5.0, end_sec=12.0))
# encode once, ask several questions
frames, times = model.load_video("my_video.mp4")
ev = model.encode_video(frames, times) # [1, 1024, 2048] video tokens
print(model.choose(None, "Is the door open at the end?", ["yes", "no"], evidence=ev))
print(model.chat(None, "What happens next?", evidence=ev))
hf download kkt94/Llama-V-JEPA-LM-Qwen3-1.7B --local-dir ./Llama-V-JEPA-LM-Qwen3-1.7B downloads the files (about 4.1 GB) if you prefer a local folder; from_pretrained accepts that folder too.
Usage notes
choosescores the options and does not generate. The prompt is"{question}\nA. {option 0}\nB. {option 1}\n...\nAnswer with the option's letter from the given choices directly."; the prediction is the argmax of the option-letter logits at the first answer position (up to 10 options, A-J). This is the protocol of the evaluation below.chatanswers a single-turn question with greedy decoding (max_new_tokens, default 256).- Chat template (
chat_template.jinja, set onmodel.tokenizerautomatically): Qwen3<|im_start|>turns in Qwen3's non-thinking format (every assistant turn starts with an empty<think></think>block), no system prompt. The video tokens come first in the single user turn, followed by the question. The API takes one video and one question per call. - Frames: 32 frames sampled uniformly over the clip (frame i at
t0 + (i + 0.5) * (t1 - t0) / 32s, nearest source frame); the clip is the whole video or[start_sec, end_sec]. Short side resized to 384 px (bicubic, antialiased), centre crop 384 x 384, ImageNet normalisation; the real frame timestamps are encoded into the video tokens. For long videos,start_sec/end_secselect the segment of interest. - Video files: any container / codec that
decordreads (mp4, webm, mkv, avi, ...). - Precision: bf16, except the bridge statistics (fp32).
rope_inv_freq_bf16(config, defaulttrue) rounds the LLM's rotary frequencies to bf16 as in the evaluation these scores come from.
| File | Contents |
|---|---|
model-*.safetensors, model.safetensors.index.json |
weights (vision encoder, bridge, language model) |
config.json |
JepaLlmConfig (model_type: jepa_llm) with the language model's config as text_config |
configuration_jepa_llm.py |
the config class |
modeling_jepa_llm.py |
JepaLlmModel: encoder, bridge, language model and the chat / choose API |
processing_jepa_llm.py |
video decoding, frame sampling, preprocessing, prompt building, chat templates |
vjepa2_1_encoder.py |
inference-only copy of the V-JEPA 2.1 encoder (MIT) |
tokenizer files, chat_template.jinja |
the language model's tokenizer and the template the model was trained with |
LICENSE_QWEN3, LICENSE_VJEPA2, LICENSE_LLAMA3_2, NOTICE_LLAMA3_2, USE_POLICY_LLAMA3_2.md |
licences (*_LLAMA3_2: Llama 3.2 licence, notice and use policy for the PLM-Video-Human FGQA training data) |
The code needs no external repository (trust_remote_code=True). Tested with Python 3.12, torch 2.13, transformers 5.10.4 (see requirements.txt).
Evaluation
Twelve scores on eleven video benchmarks for Llama-V-JEPA-LM-Qwen3-1.7B and every open model of the 2B class. All models are evaluated by the same harness with one protocol: 32 frames, the same items and prompts, option-letter scoring. Bold: best in the table.
| Model | MVB | PT | TVB | TC | TOM | MoB | NQA | MVP | TB | SSv2 | VG-T | VG-V | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-V-JEPA-LM-Qwen3-1.7B | 69.5 | 75.3 | 70.7 | 73.5 | 50.9 | 54.3 | 74.0 | 54.6 | 63.9 | 91.6 | 63.2 | 50.8 | 66.02 |
| Qwen3-VL-2B | 61.0 | 65.1 | 49.8 | 67.4 | 30.3 | 57.5 | 77.1 | 32.3 | 65.9 | 59.2 | 36.2 | 27.6 | 52.45 |
| InternVL3.5-2B | 66.8 | 64.2 | 50.9 | 65.7 | 31.0 | 56.2 | 76.1 | 28.8 | 71.3 | 57.0 | 40.0 | 21.0 | 52.42 |
| PLM-1B | 68.0 | 69.8 | 48.2 | 60.5 | 24.5 | 51.7 | 79.8 | 30.9 | 58.1 | 61.7 | 33.2 | 29.6 | 51.33 |
| Apollo-1.5B | 61.1 | 63.9 | 47.9 | 60.3 | 27.8 | 51.1 | 78.8 | 26.0 | 55.8 | 56.0 | 38.8 | 22.4 | 49.16 |
| VideoLLaMA3-2B | 57.9 | 62.9 | 44.0 | 64.1 | 24.0 | 56.3 | 75.4 | 23.1 | 64.1 | 50.0 | 38.0 | 25.2 | 48.75 |
| Gemma 4 E2B | 39.8 | 46.3 | 36.4 | 51.5 | 18.7 | 41.1 | 51.4 | 8.1 | 61.4 | 37.0 | 24.0 | 22.4 | 36.51 |
MVB: MVBench, PT: PerceptionTest, TVB: TVBench, TC: TempCompass, TOM: TOMATO, MoB: MotionBench, NQA: NExT-QA, MVP: Minimal Video Pairs (pair accuracy), TB: TemporalBench (binary), SSv2: Something-Something v2 (multiple choice), VG-T / VG-V: Vinoground text / video (pair accuracy), Avg: mean of the 12 scores.
Held-out generalization. Six construction sets built from videos and settings never used in training (unseen playback-speed ratios, a new source corpus for speed and for direction, unseen mirror × reversal and acceleration × reversal combinations, camera motion from an unseen simulator; chance 33.3): 95.0 average for Llama-V-JEPA-LM-Qwen3-1.7B, vs. 43.8 for Qwen3-VL-2B.
Inference cost per 32-frame clip on one H100 (batch 1, bf16, video decoding excluded; times: median of 10 runs after 3 warm-ups; 480p clip unless noted):
| Model | Visual tokens | Encoder (TFLOPs) | LLM prefill (TFLOPs) | LLM prefill (ms) | First token, 480p (ms) | Decode (ms / token) | First token, 720p (ms) |
|---|---|---|---|---|---|---|---|
| Llama-V-JEPA-LM-Qwen3-1.7B | 1,024 | 13.9 | 3.3 | 22 | 83 | 20.0 | 82 |
| Qwen3-VL-2B | 1,920 / 11,520 | 5.0 | 7.0 | 31 | 66 | 18.2 | 308 |
| InternVL3.5-2B | 8,192 | 23.2 | 40.5 | 104 | 159 | 18.4 | 159 |
| PLM-1B | 8,192 | 22.2 | 25.2 | 57 | 146 | 9.3 | 146 |
First token = vision encoder + connector + LLM prefill. Decode: per generated token with KV cache. --: not timed. The visual-token count and cost of Llama-V-JEPA-LM models do not depend on the input resolution. The encoder time is a property of the encoder: V-JEPA 2.1 attends jointly over all 32 frames (9,216 patches), which is what carries the dynamics, while the open models use frame-wise image encoders (SigLIP-2-based in Qwen3-VL, InternViT, PE) that attend within one frame or frame pair at a time; it is identical for every Llama-V-JEPA-LM model on the same encoder.
Training
Data. 4.60M samples drawn at fixed per-source weights from 175 public sources: open video caption and QA collections (LLaVA-Video, ShareGPTVideo, PerceptionLM data, Charades-Ego, EPIC-Kitchens, HoloAssist, SSv2, Kinetics-700, SA-V, ActivityNet and others) plus 33 counterfactual sources (509,193 rows, about 6% of the drawn samples). The counterfactual rows come from 14 constructions in 6 families. Each takes a clip of a public dataset and a transformation; the answer follows from the transformation, so no new human annotation and no language model are involved. Within each source every scene appears with each answer equally often in expectation.
| Family | Constructions | Rows |
|---|---|---|
| Sign flip | clip reversed or mirrored, camera move rendered or reversed, two clips each optionally reversed | 117,066 |
| Order | two events joined in sequence or stacked; which segment matches a two-action caption | 85,579 |
| Position | several clips joined, one holds the queried action: beginning / middle / end | 48,386 |
| Magnitude | relative speed, acceleration / deceleration, slow motion vs. time-lapse | 141,084 |
| Count | one action repeated k times; number of distinct activities | 35,614 |
| Stop/cut | clip cut before the action completes; physically impossible simulator edits | 81,464 |
Recipe. The V-JEPA 2.1 encoder stays frozen. After a bridge-only warm-up (0.6–0.8% of the steps), the bridge and the full language model are trained together in one continuous stage: 27,382 steps at global batch 168, learning rate 1.05e-5 for the language model and 2e-4 for the bridge, cosine decay, weight decay 0.01 / 0.05, maximum sequence length 2,560 tokens.
Hardware. 14 NVIDIA H100 GPUs (12 clips per GPU), 16.4 hours, 229 H100-hours in total; training compute 121 EFLOPs (including the frozen encoder's forward pass).
Training data
Fine-tuned on the training splits of public video-language datasets; no new human annotation.
| Dataset | License | Note |
|---|---|---|
| Something-Something v2 | Qualcomm research-use license | non-commercial |
| LLaVA-Video-178K | Apache-2.0 (card: academic research) | GPT-4o-generated |
| ShareGPTVideo (LLaVA-Hound) | Apache-2.0 | GPT-generated |
| HoloAssist | CDLA-Permissive-2.0 | – |
| SA-V | CC BY 4.0 | – |
| PLM-Video-Human (RCap, RDCap, RTLoc) | CC BY 4.0 | – |
| PLM-Video-Human FGQA | Llama 3.2 Community License | Llama 3.2 output |
| Charades / Charades-Ego | AI2 non-commercial license | non-commercial |
| EPIC-KITCHENS-100 | CC BY-NC 4.0 | non-commercial |
| EgoDex | CC BY-NC-ND 4.0 | non-commercial |
| Kinetics-700 | CC BY 4.0 (annotations) | YouTube videos |
| NVIDIA PhysicalAI Physical-Interaction-Scenes | OpenMDW-1.1 | – |
| RoboVQA | CC BY 4.0 | – |
| Language-Table | Apache-2.0 | – |
| CLEVRER | see project page | – |
| TimeIT | CC BY 4.0 (instructions) | source videos under their own terms |
| PerceptionTest (train) | CC BY 4.0 | – |
| STAR (train) | Apache-2.0 | Charades videos |
| Oops! | CC BY-NC-SA 4.0 | non-commercial |
| HT-Step | CC BY-NC 4.0 | non-commercial |
| Others (ActivityNet, YouCook2, LSMDC, Ego-QA, Video-Holmes, VidLN, Countix) | see each dataset | research use |
| UltraChat-200k prompts | MIT | answers generated by the base LLM |
| Reasoning / answer rewrites | generated by Qwen3-8B / Qwen3-VL-8B (Apache-2.0) | – |
Each dataset remains under its own terms; this release grants no rights in them.
Intended use
- Intended: question answering about and description of videos in English: actions, events and their order, motion direction and speed, camera motion, counting, and state changes; research on video-language models and world-model encoders.
- Out of scope: image, document, chart or OCR question answering, which were not training targets; use that violates the Llama 3.2 Acceptable Use Policy (which applies to the FGQA training-data component) or applicable law.
Several training datasets restrict commercial use. We release the weights for research and non-commercial use; anyone considering commercial use is responsible for checking the terms of the training data.
Limitations
- Single images and text in frames. V-JEPA 2.1 is trained only on video, to predict masked spatiotemporal regions in latent space. It is never trained to align with text or to read it. Text recognition (OCR, documents, charts, slides) and single-image question answering are therefore outside what the encoder was built for, and they are not the target of this model. It is built for how scenes change over time. For text-heavy or single-image tasks, an image–text encoder is the natural choice.
- Long videos. The model targets clip-level understanding of dynamics, 32 frames per clip. Hour-scale long-video benchmarks need many clips per video and far longer contexts, in training as well as at inference. At the scale of this release (1B–30B, one training stage of a few hundred H100-hours), that training cost is impractical, so we did not train or evaluate for it. For a long video,
start_sec/end_secselect the segment to ask about.
License
This model combines components under the following licences:
- Language model: Apache License 2.0. The language-model weights are a modified version of Qwen3-1.7B (
LICENSE_QWEN3): all weights were fine-tuned on video-language data, and video tokens from the bridge are inserted into its input embeddings. The tokenizer files are unmodified copies from Qwen3-1.7B. - Video encoder: MIT License. The encoder weights are the V-JEPA 2.1 ViT-L checkpoint distilled from ViT-G,
vjepa2_1_vitl_dist_vitG_384.ptreleased by Meta with facebookresearch/vjepa2 (MIT,LICENSE_VJEPA2), unchanged (frozen).vjepa2_1_encoder.pyis adapted from that repository under the MIT License. - Bridge weights and model code are released under the Apache License 2.0.
- Training data component: Llama 3.2 Community License. The training data include the FGQA split of PLM-Video-Human, an output of Llama 3.2 that is subject to the Llama 3.2 Community License (
LICENSE_LLAMA3_2) and the Llama 3.2 Acceptable Use Policy (USE_POLICY_LLAMA3_2.md); as that license requires for models trained on it, the model name begins with "Llama". Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. (NOTICE_LLAMA3_2) - Training datasets remain under their own terms (see Training data).
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP, AI Computing Support Project for R&D) grant funded by the Korea government (MSIT) (No. RS-2026-25505492, High-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale).
Citation
@misc{kim2026vjepalm,
title = {V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations},
author = {Kim, Keuntae and Kim, Dongjun and Choi, Yong Suk},
year = {2026},
eprint = {2611.17831},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2611.17831}
}
Please also cite V-JEPA 2 / V-JEPA 2.1 and Qwen3.
Model family
All sizes are in the Llama-V-JEPA-LM collection.
| Model | Language model | Encoder | Avg (12) |
|---|---|---|---|
| Llama-V-JEPA-LM-Llama3.2-1B | Llama-3.2-1B-Instruct | V-JEPA 2.1 ViT-L | 62.21 |
| Llama-V-JEPA-LM-Qwen3-1.7B (this model) | Qwen3-1.7B | V-JEPA 2.1 ViT-L | 66.02 |
| Llama-V-JEPA-LM-Llama3.2-3B | Llama-3.2-3B-Instruct | V-JEPA 2.1 ViT-g | 68.97 |
| Llama-V-JEPA-LM-Qwen3-4B | Qwen3-4B-Instruct-2507 | V-JEPA 2.1 ViT-g | 72.14 |
| Llama-V-JEPA-LM-Llama3.1-8B | Llama-3.1-8B-Instruct | V-JEPA 2.1 ViT-g | 70.51 |
| Llama-V-JEPA-LM-Qwen3-8B | Qwen3-8B | V-JEPA 2.1 ViT-g | 70.11 |
| Llama-V-JEPA-LM-Qwen3-30B-A3B | Qwen3-30B-A3B | V-JEPA 2.1 ViT-G | 69.49 |
Model card authors
Keuntae Kim, Hanyang University, Artificial Intelligence Lab