CLI-0.5B: Cross-Layer Injection for Deep Vision-Language Fusion

Official model checkpoint of "From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion" (ECCV 2026).

CLI-0.5B is the 0.5B-parameter variant of our Cross-Layer Injection (CLI) framework, built on top of LLaVA-OneVision-Qwen2-0.5B (Qwen2-0.5B LLM + SigLIP-so400m-patch14-384 vision encoder). It replaces the conventional one-to-one vision–language bridge with a dynamic many-to-many connection, allowing every designated LLM decoder layer to query the full visual hierarchy on demand.

Abstract. Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection that links only the output of the vision encoder to the input of the large language model (LLM). This static architecture fundamentally limits the ability of LLMs to achieve comprehensive alignment with hierarchical visual knowledge, compromising their capacity to accurately integrate local details with global semantics into coherent reasoning. To resolve this, we introduce Cross-Layer Injection (CLI), a novel framework that forges a dynamic "many-to-many" bridge between the two modalities. CLI consists of two synergistic, parameter-efficient components: an Adaptive Multi-Projection (AMP) module that harmonizes features from diverse vision layers, and an Adaptive Gating Fusion (AGF) mechanism that empowers the LLM to selectively inject the most relevant visual information based on its real-time decoding context. We validate the effectiveness and versatility of CLI by integrating it into LLaVA-OneVision and LLaVA-1.5. Extensive experiments on 28 diverse benchmarks demonstrate significant performance improvements, establishing CLI as a scalable paradigm that unlocks deeper multimodal understanding by granting LLMs on-demand access to the full visual hierarchy. Code is available at https://github.com/codefuse-ai/CLI.

Key Features

• Many-to-Many Fusion — CLI replaces the conventional one-to-one vision–language bridge with a dynamic many-to-many architecture; each LLM decoder layer can query the full visual hierarchy on demand.
• Adaptive Multi-Projection (AMP) — Parameter-efficient LoRA-based projectors that harmonize features from diverse vision encoder layers into a shared semantic space.
• Adaptive Gating Fusion (AGF) — A query-based attention gate that dynamically selects and injects the most relevant visual information based on the LLM's real-time decoding context.
• Architecture-Agnostic — Validated on both LLaVA-OneVision (0.5B/7B) and LLaVA-1.5-7B. This checkpoint is the 0.5B variant.
• Minimal Overhead — Only ~1.3% inference memory increase with marginal latency impact.

Model Details

This model card is a model checkpoint hosted here for convenience. To actually load and run CLI-0.5B you need the custom model implementation (AMP + AGF) shipped with the codefuse-ai/CLI codebase — a plain transformers LlavaOneVision load will not reproduce CLI behavior.

How to Use

CLI requires the custom architecture from the codefuse-ai/CLI repository. The layer configuration passed at load time must match the training configuration above.

1. Install

git clone https://github.com/codefuse-ai/CLI.git
cd CLI

conda create -n cli python=3.10 -y
conda activate cli

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

2. Download this checkpoint

# Option A: huggingface-cli
huggingface-cli download codefuse-ai/CLI-0.5B --local-dir ./checkpoints/CLI-0.5B

# Option B: git-lfs
git clone https://huggingface.co/codefuse-ai/CLI-0.5B ./checkpoints/CLI-0.5B

3. Inference

The way to run CLI-0.5B is through the repo's evaluation/inference entry point, which instantiates the AMP + AGF modules and wires the cross-layer connections using the RANGE-1-24-4 / RANGE-1-28-4 configuration:

python run_eval_cli.py \
    --model_path ./checkpoints/CLI-0.5B \
    --eval_tasks mme,mmmu,ai2d \
    --vlm_exp_layers RANGE-1-24-4 \
    --vision_exp_layers RANGE-1-28-4 \
    --num_gpus 1

⚠️ Important: The --vlm_exp_layers RANGE-1-24-4 and --vision_exp_layers RANGE-1-28-4 flags are mandatory — they rebuild the CLI architecture (which LLM layers receive injections, which vision layers are extracted). Loading with the default one-to-one projector path or with transformers.LlavaOnevisionForConditionalGeneration will not reproduce CLI.

For single-turn visual chat / custom inference, use the LLaVA-OneVision conversation template (qwen_1_5 prompt version) with anyres_max_9 image preprocessing. See run_eval_cli.py and llava/eval/ in the repo for reference implementations.

Evaluation

We evaluate on 28 benchmarks across three categories using the LMMs-Eval framework:

• Chart / Diagram / Document: ai2d, chartqa, docvqa_val/test, infovqa_val/test
• Perception & Reasoning: mme, mmbench_en_dev, mmvet, mmmu, mmstar, mathvista_testmini, mathverse_*, gqa, ok_vqa, scienceqa_img, seedbench, pope
• Real-world & Visual Chat: realworldqa, llava_in_the_wild

Run all benchmarks in one command:

python run_eval_cli.py \
    --model_path ./checkpoints/CLI-0.5B \
    --eval_tasks ai2d,chartqa,docvqa_val,docvqa_test,infovqa_val,infovqa_test,mme,mmbench_en_dev,mmvet,mmmu,mmstar,mathvista_testmini,gqa,ok_vqa,scienceqa_img,seedbench,pope,realworldqa,llava_in_the_wild \
    --vlm_exp_layers RANGE-1-24-4 \
    --vision_exp_layers RANGE-1-28-4 \
    --num_gpus 1

Citation

If you find this work useful, please cite:

@inproceedings{chen2026cli,
    title={From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion},
    author={Chen, Cheng and Guo, Yuyu and Zeng, Pengpeng and Song, Jingkuan and Di, Peng and Yu, Hang and Gao, Lianli},
    booktitle={European Conference on Computer Vision (ECCV)},
    year={2026}
}

Acknowledgements

This checkpoint is built upon LLaVA-OneVision-Qwen2-0.5B, SigLIP, and the LLaVA-NeXT framework, and is evaluated with LMMs-Eval.

License

Released under the Apache 2.0 License.