CoVisco-L-14 (Vision Encoder)

CoVisco-L-14 is the Stage 1 vision encoder of the CoVisco pipeline (Codec-Native Vision Encoder with Native Token Compression for Efficient Unified Visual Understanding). It encodes images and videos in a unified way and is trained by SigLIP-style distillation against two teacher branches.

This repository releases only the vision-encoder weights in the safe .safetensors format, together with the minimal model code and a self-contained loader. The model is a transformers.PreTrainedModel and is loadable via from_pretrained (see below).

Contents

• Architecture · Files
• Loading the model · Quick start
• Usage: image · native resolution · video · codec filtering
• Model interface · token-level features
• Classification and retrieval · Notes

Architecture

• Backbone: ViT-L/14, image_size=224, patch_size=14, layers=24, width=1024, heads=16
• Native token compression: num_query_per_seg=100 learnable abstract tokens, segment_t_size=32
• 4D RoPE (rope_theta=10000) for spatio-temporal positions
• Three projection heads (each LayerNorm + Linear, output L2-normalized):
  • to_image_caption → 1536-d, SigLIP2 space (image classification / retrieval)
  • to_image → 1536-d
  • to_video_caption → 4096-d, Qwen3-VL-Embedding space (video classification / retrieval)
• Independent logit_scale / logit_bias per contrast pair

Files

• model.safetensors — model weights (fp32)
• config.json — encoder config + head dims + preprocessing config; includes architectures / model_type / auto_map so the repo is HF-loadable
• covisco_vit.py, covisco_model.py, reconstruction_decoder.py — model code (covisco_model.py also provides the build_preprocess / encode_image / encode_video convenience methods, available on any from_pretrained model)
• load_covisco.py — CLI + thin functional wrappers over those model methods
• codec_visidx.py — codec-based token filtering (see below)
• requirements.txt, LICENSE

Loading the model

There are two ways to load, and they return the same weights and the same model class (CoViscoModel) — they differ only in where the files come from.

1. from_pretrained (recommended) — standard HF API, downloads from the Hub:

from transformers import AutoModel

REPO = "ernie-research/CoVisco-L-14"
model = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval().cuda()
# equivalently, import the class directly (no trust_remote_code needed):
# from covisco_model import CoViscoModel
# model = CoViscoModel.from_pretrained(REPO).eval().cuda()

trust_remote_code=True is required so HF loads the modeling code shipped in this repo via auto_map. The loaded model carries the convenience methods listed below, so a Hub user gets the full image/video pipeline without needing load_covisco.py.

2. build_model (local files) — the helper in load_covisco.py, for when you have cloned/downloaded the repo:

from load_covisco import build_model

model, cfg = build_model("config.json", "model.safetensors",
                         device="cuda", segment_t_size=16)   # overrides optional

build_model is literally CoViscoModel.from_pretrained(<dir of config.json>) plus optional runtime overrides (segment_t_size / image_size), and it also returns the raw cfg dict parsed from config.json. Prefer it only for the CLI or scripted local runs; otherwise use from_pretrained.

Preprocessing & encoding helpers (methods on the model)

The only knobs a user adjusts are the resolution and (for video) segment_t_size. The model exposes everything as methods (reading model.config.preprocess_cfg), so a from_pretrained model is self-sufficient:

• model.build_preprocess(size=None) — square per-frame transform (shortest-edge resize + center crop).
• model.build_native_preprocess(max_side=1400) — aspect-preserving transform at (near-)native resolution.
• model.encode_image(image, preprocess=None, branch=...) — single image (path / PIL / tensor) → (1, D) (preprocess defaults to build_preprocess()).
• model.embed_images(inputs, ...) — batched: inputs is a list of path/PIL/tensor items or a (B, C, H, W) tensor → (B, D).
• model.read_video_frames(path, num_frames) (needs PyAV) + model.encode_video(frames, preprocess=None, visible_indices=None, branch=...) — single clip → (1, D).
• model.embed_videos(inputs, ...) — batched: inputs is a list of clips (each a frame sequence or (T, C, H, W) tensor) or a (B, C, T, H, W) tensor.
• model.set_segment_t_size(16) — set the temporal segment size (runtime-only; video frame count T must be a positive multiple of it).

The same per-frame transform serves both images and video; the only extra step for video is sampling frames and stacking them into a (B, C, T, H, W) clip, which encode_video does for you. The load_covisco.py functions (build_preprocess, encode_image, …) are thin wrappers over these methods, kept for local/CLI use.

Quick start

Install the deps and run a 4-line image example — downloads from the Hub, no clone needed:

pip install torch torchvision pillow "transformers>=4.37"

from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("ernie-research/CoVisco-L-14",
                                  trust_remote_code=True).eval().cuda()  # .cpu() if no GPU
emb = model.encode_image(Image.open("your.jpg"))   # (1, 1536), L2-normalized

See Programmatic usage for branches/resolution, Video usage for clips, and Model interface for the raw forward.

CLI (requires cloning this repo)

pip install -r requirements.txt
# smoke test (blank 224x224 image)
python load_covisco.py
# real image, square 224 preprocessing
python load_covisco.py --image your.jpg
# native resolution (keeps aspect ratio, long side <= 1400 px)
python load_covisco.py --native_resolution --image your.jpg
# native resolution with a smaller memory cap
python load_covisco.py --native_resolution --max_side 896 --image your.jpg

CLI flags: --config, --weights, --image, --device, --native_resolution, --max_side, --image_size, --segment_t_size.

Programmatic usage (image)

from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("ernie-research/CoVisco-L-14",
                                  trust_remote_code=True).eval().cuda()
preprocess = model.build_preprocess(size=224)   # square 224; size optional
emb = model.encode_image(Image.open("your.jpg"), preprocess)
# emb: [1, 1536], L2-normalized (SigLIP2 branch, to_image_caption)

# 4096-d Qwen3-VL-Embedding-space embedding instead. For images this head is
# keyed `to_image_video_caption` (it reuses the video-caption head weights;
# `to_video_caption` only exists under modality="video").
emb = model.encode_image(Image.open("your.jpg"), preprocess,
                         branch="to_image_video_caption")

preprocess is optional — model.encode_image(img) falls back to model.build_preprocess() (square, default size from config.preprocess_cfg).

Batching

embed_images encodes many images at once and returns (B, D). Items can be paths, PIL images, or (C, H, W) tensors; or pass a ready (B, C, H, W) tensor and skip preprocessing entirely (do it outside):

import torch
from PIL import Image

paths = ["a.jpg", "b.jpg", "c.jpg"]

# from files / PIL (square preprocessing, applied inside)
embs = model.embed_images(paths)                                # (3, 1536)

# bring-your-own preprocessing: build the batch tensor yourself, then embed
pp = model.build_preprocess(size=224)
px = torch.stack([pp(Image.open(p).convert("RGB")) for p in paths])  # (B,C,H,W)
embs = model.embed_images(px)                                   # (B, 1536)

Dense batching needs a uniform (C, H, W), so use the square build_preprocess. Native/variable-resolution images cannot be stacked — encode those one at a time with encode_image. (The raw forward model(px, modality="image")["to_image_caption"] is equivalent and also batch-native.)

Native resolution (images)

The encoder is RoPE-only (no learned position table) and derives its patch grid as h = H // 14, w = W // 14 at runtime, so it accepts any H x W that is a multiple of the patch size — including non-square inputs:

from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("ernie-research/CoVisco-L-14",
                                  trust_remote_code=True).eval().cuda()
preprocess = model.build_native_preprocess(max_side=1400)
emb = model.encode_image(Image.open("doc_page.png"), preprocess)

build_native_preprocess keeps the original aspect ratio and only scales the long side down to max_side (memory cap) and rounds each side to a multiple of patch_size. This is the recommended mode for text-heavy document pages, where the square 224 center-crop destroys detail.

Because a batch is a single dense tensor (no resolution packing), encode one image at a time when using native resolution.

Video usage

Videos are passed as a (B, C, T, H, W) tensor (channel before time). T must be a positive multiple of segment_t_size; the encoder splits the clip into T // segment_t_size temporal segments. segment_t_size is a pure runtime parameter (RoPE positions are computed from the runtime frame grid; no weight shape depends on it), so set it on the model — no manual config rebuild needed. encode_video samples/stacks the frames for you:

from transformers import AutoModel

NUM_FRAMES = 16      # frames uniformly sampled per clip
SEGMENT_T_SIZE = 16   # frames per temporal segment (must divide NUM_FRAMES;
                      # 16 -> 1 segment here; config.json default is 32)
IMAGE_SIZE = 224      # square resolution per frame (any multiple of 14 works)

model = AutoModel.from_pretrained("ernie-research/CoVisco-L-14",
                                  trust_remote_code=True).eval().cuda()
# segment_t_size is runtime-only (RoPE, no weight shape depends on it):
# same weights, override the config.json default (32).
model.set_segment_t_size(SEGMENT_T_SIZE)

# Same per-frame transform as images. For aspect-preserving native-resolution
# frames use model.build_native_preprocess(max_side=1400) instead.
preprocess = model.build_preprocess(size=IMAGE_SIZE)

# read_video_frames needs PyAV (optional): pip install av
frames = model.read_video_frames("video.mp4", NUM_FRAMES)   # list of PIL images
video_emb = model.encode_video(frames, preprocess)
# video_emb: (1, 4096), L2-normalized (to_video_caption, Qwen3-VL-Embedding space)

encode_video applies the shared per-frame preprocess, stacks the frames into the (B, C, T, H, W) tensor the encoder expects, and runs the forward pass. If you already have a (T, C, H, W) clip tensor you can pass it directly instead of a list of PIL frames.

For multiple clips, use embed_videos → (B, 4096). Pass a list of clips (each a frame sequence or a (T, C, H, W) tensor) or a ready (B, C, T, H, W) tensor; all clips must share (C, T, H, W):

clips = [model.read_video_frames(p, NUM_FRAMES) for p in ["a.mp4", "b.mp4"]]
video_embs = model.embed_videos(clips, preprocess)          # (2, 4096)

A frame count that is not a multiple of the effective segment_t_size raises an AssertionError at forward time; a clip shorter than segment_t_size is also accepted (single segment, T frames) — positions stay consistent because RoPE is computed from the runtime grid.

Codec-based token filtering (video)

The encoder natively supports sparse patch selection: passing visible_indices (visidx) makes get_segments compute only the selected patches and derive 4D-RoPE from them — a codec-style token budget without any mosaic packing. codec_visidx.py generates these indices from video codec signals, with the same scoring math as the training-time precompute_codec_visidx pipeline:

• Backend 1 — cv_reader (exact, matches training): install Compressed-Video-Reader (C++ extension + patched FFmpeg) to read H.264/HEVC macroblock motion vectors and DCT residuals. The scoring functions are embedded verbatim, so the visidx is numerically identical to the training-time precompute.
• Backend 2 — PyAV (fallback, no extra deps): FFmpeg AV_FRAME_DATA_MOTION_VECTORS side data for motion; luminance frame differences approximate the residual.
• Selection (both backends): per-segment equal top-K (keep_ratio of segment_t_size * patches_per_frame), first frame of each segment always kept in full, indices sorted ascending — the layout get_segments requires.

Two usage modes:

End-to-end (scoring runs inline, right before the forward pass):

from transformers import AutoModel
from codec_visidx import compute_codec_visidx, sample_frame_ids, _decode_frames

# Single source of truth — the SAME segment_t_size / resolution must reach
# every call below, so configure them explicitly (no hidden defaults).
NUM_FRAMES, SEGMENT_T_SIZE, IMAGE_SIZE, KEEP_RATIO = 64, 16, 224, 0.57

visidx, meta = compute_codec_visidx("video.mp4", num_frames=NUM_FRAMES,
                                    segment_t_size=SEGMENT_T_SIZE,
                                    image_size=IMAGE_SIZE, keep_ratio=KEEP_RATIO)

model = AutoModel.from_pretrained("ernie-research/CoVisco-L-14",
                                  trust_remote_code=True).eval().cuda()
model.set_segment_t_size(SEGMENT_T_SIZE)
preprocess = model.build_preprocess(size=IMAGE_SIZE)   # must equal the image_size above

# meta records what visidx was ACTUALLY computed for — assert the model and
# preprocess agree, so a mismatched segment_t_size / resolution fails loudly
# instead of silently misaligning patch indices.
assert meta["segment_t_size"] == model.encoder.segment_t_size, "segment_t_size mismatch"
assert meta["image_size"] == IMAGE_SIZE, "image_size mismatch"

# Decode the SAME frames the scoring used. sample_frame_ids samples over
# meta["total_frames"] (the probed stream length), so this guarantees patch
# index i still refers to the same patch — don't substitute read_video_frames
# here, it samples over the decoded-frame count instead.
_, rgbs, *_ = _decode_frames("video.mp4",
                             sample_frame_ids(meta["total_frames"], NUM_FRAMES))

# Pass visidx into encode_video to run in codec (sparse-patch) mode.
video_emb = model.encode_video(rgbs, preprocess, visible_indices=visidx)   # (1, 4096)

codec_visidx is a local module shipped in this repo (not downloaded by trust_remote_code), so clone/download the repo to use codec mode.

Pre-extracted (batch datasets; output layout matches the SFT precompute_codec_visidx pipeline):

# videos.jsonl: one {"video": "/path.mp4", "key": "unique_id"} per line
python codec_visidx.py --jsonl videos.jsonl --out_root cache/ \
    --num_frames 64 --segment_t_size 16 --keep_ratio 0.57
# writes cache/<key>/{visidx.npy, frame_ids.npy, meta.json}

import json
import numpy as np
from transformers import AutoModel
from codec_visidx import load_visidx, _decode_frames

# meta.json is the single source of truth for this cache entry — read
# segment_t_size / image_size from it instead of re-typing them, so the model
# and preprocess cannot drift from what the visidx was computed for.
meta = json.load(open("cache/<key>/meta.json"))

model = AutoModel.from_pretrained("ernie-research/CoVisco-L-14",
                                  trust_remote_code=True).eval().cuda()
model.set_segment_t_size(meta["segment_t_size"])
preprocess = model.build_preprocess(size=meta["image_size"])

# Reuse the exact frame ids saved alongside the visidx.
frame_ids = np.load("cache/<key>/frame_ids.npy").tolist()
_, rgbs, *_ = _decode_frames(meta["video"], frame_ids)
video_emb = model.encode_video(rgbs, preprocess,
                               visible_indices=load_visidx("cache/<key>"))

Constraints: the clip fed to encode_video must use the same num_frames linspace sampling, the same square image_size preprocessing, and the same segment_t_size as the visidx computation (meta.json records these for verification). With keep_ratio=0.57 and a 224² grid, each 16-frame segment keeps ~2329 of its 4096 patches (~57%), cutting encoder FLOPs correspondingly.

Model interface

The helpers above all wrap a single call to the model's forward. Call it directly when you need full control (custom batching, both modalities at once, token-level outputs):

out = model(
    pixel_values,                 # image: (B, C, H, W);  video: (B, C, T, H, W)
    modality="image",            # "image" | "video" | None (compute all heads)
    visible_indices=None,         # optional (B, L) codec sparse-patch indices
    return_intermediates=False,   # also return query_tokens / vit_tokens
    return_attentions=False,      # also return per-layer attentions (needs eager attn)
    run_decoder=False,            # keep False — no reconstruction decoder in this release
)

Arguments:

• pixel_values — an image batch (B, C, H, W) or a video batch (B, C, T, H, W) (channel before time). Build it with build_preprocess / build_native_preprocess; every sample in a batch must share the same spatial (and, for video, temporal) size. For video, T must be a positive multiple of segment_t_size.
• modality — selects which projection heads run: "image" → to_image / to_image_caption / to_image_video_caption, "video" → to_video_caption only, None → all.
• visible_indices — optional (B, L) codec indices (see Codec filtering).
• return_intermediates / return_attentions — add the pre-pooling token outputs / attention weights to the returned dict.

Minimal embedding straight from the forward (equivalent to embed_images / embed_videos, no convenience method involved):

import torch

# image batch -> px (B, C, H, W)
pp = model.build_preprocess()
px = torch.stack([pp(im.convert("RGB")) for im in images]).to(model.device)
emb = model(px, modality="image", run_decoder=False)["to_image_caption"]   # (B, 1536)

# video batch -> px_video (B, C, T, H, W)
vid = model(px_video, modality="video", run_decoder=False)["to_video_caption"]  # (B, 4096)

The call returns a dict:

Note the 4096-d image head is keyed to_image_video_caption (not to_video_caption) even though it shares the same projection weights; to_video_caption is only produced under modality="video".

Extracting token-level representations

Pass return_intermediates=True to also get the pre-pooling token outputs. This is what the CoVisco SFT stage consumes as visual inputs (e.g. as the vision tower of an LLM). encode_image / encode_video accept the same flag and then return the full forward dict instead of a single embedding:

from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("ernie-research/CoVisco-L-14",
                                  trust_remote_code=True).eval().cuda()

# return_intermediates=True -> full dict (branch embeddings + token outputs).
out = model.encode_image(Image.open("your.jpg"), return_intermediates=True)
# non-default resolution: pass model.build_preprocess(size=336) as 2nd arg.

out["to_image_caption"]   # (B, 1536)          the embedding (still present)
out["pooled_output"]      # (B, 1024)          pooled representation
out["query_tokens"]       # (B, 1, 100, 1024)  per-segment abstract/query tokens
out["vit_tokens"]         # (B, 1, 256, 1024)  per-segment ViT patch tokens
# 224x224 / patch 14 -> 16x16 = 256 patches per frame/segment
# (the first 100 tokens of every segment are the learnable query tokens)

# Equivalent low-level call if you prefer to build the tensor yourself:
# px = model.build_preprocess()(Image.open("your.jpg")).unsqueeze(0).cuda()
# out = model(px, modality="image", run_decoder=False, return_intermediates=True)

For video the layout is (B, S, ...) with S = T // segment_t_size segments, e.g. 16 frames at segment_t_size=16 (one segment of 16 frames) → query_tokens (B, 1, 100, 1024) and vit_tokens (B, 1, 16*256, 1024) (256 patches per frame, all frames of the segment).

Notes:

• The projection heads consume pooled_output; query_tokens / vit_tokens are the uncompressed sequence outputs if you need token-level features.
• return_attentions=True additionally returns per-layer attention weights (requires eager attention, not flash-attn).

Classification and retrieval

This repo ships only the vision tower. To run zero-shot classification or text↔image/video retrieval you must match embeddings against the frozen teacher text encoder of the corresponding branch:

• to_image_caption (1536-d) ↔ SigLIP2 text tower (timm/ViT-gopt-16-SigLIP2-384, pretrained=webli)
• to_video_caption (4096-d) ↔ Qwen/Qwen3-VL-Embedding-8B text encoder

The embedding spaces are branch-specific and not interchangeable.

Notes

• The encoder is stored twice in the state dict, under encoder.* and its alias visual.* (the model sets self.visual = self.encoder). Both resolve to the same module; a clean load reports no missing/unexpected keys. This is why the file is ~2.4 GB for a ~320 M-parameter encoder.
• image_size in config.json is informational only — it does not affect any weight shape or forward computation (patch embedding is a conv and positions are RoPE), so the same weights run at any resolution.
• flash_attn is optional; the code falls back to standard attention if it is not installed.
• use_reconstruction=False for the released weights (no decoder parameters); pass run_decoder=False to skip the decoder path entirely.
• The training-time data-augmentation paths (uniform_sample_frames / frame_concat) have been removed from this release; inference always uses the get_segments path with the frame count and segment_t_size you configure.

License

MIT License.

Citation

@article{liu2026covisco,
  title={CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding},
  author={Liu, Yulong and Han, Xiaotian and Shang, Junyuan and Ding, Yuchen and Zhang, Zhenyu and Wang, Shuohuan and Zhu, Guibo and Han, Sirui and Yu, Dianhai},
  journal={arXiv preprint arXiv:2609.39924},
  year={2026}
}