HoloCT-8B

A 3D foundation model unifying CT interpretation and segmentation

HoloCT-8B is the larger model in the HoloCT family, built around a Qwen3-VL-8B language backbone. It takes a 3D CT volume and a text prompt to perform lesion screening, visual question answering, report generation and segmentation. Grounded responses link textual findings to voxel-level masks.

HoloCT combines a Med3DVLM-derived global encoder, a VISTA3D-derived detailed encoder, a language backbone and a 3D mask decoder to connect whole-volume understanding with fine-grained spatial information.

Key features

• Unified CT interpretation: lesion screening, visual question answering and report generation through text prompts.
• Grounded text generation: produce textual findings and their corresponding 3D masks.
• Language-guided segmentation: segment anatomical targets and findings described in free text.
• Paired inference: compare grounded and ungrounded responses on the same CT with the supplied demos.

Model details

Quick start

Use Ubuntu 22.04, Python 3.10.16 and an NVIDIA GPU supporting bfloat16. The HoloCT demos load the checkpoint through the project's model and CT processor classes.

1. Install

git clone https://github.com/alibaba-damo-academy/HoloCT.git
cd HoloCT

conda env create -f environment.yml
conda activate holoct
python -m pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124
python -m pip install -r requirements.txt

2. Download HoloCT-8B

python -c "from huggingface_hub import snapshot_download; snapshot_download('Alibaba-DAMO-Academy/HoloCT-8B', local_dir='weights/HoloCT-8B')"

3. Run on your CT

Set --ct to a 3D NIfTI CT in Hounsfield units. This example compares an ungrounded answer with a grounded answer for the same volume:

CUDA_VISIBLE_DEVICES=0 python -m inference.scripts.demo_vqa \
  --model weights/HoloCT-8B \
  --ct demo_ct/my_ct.nii.gz \
  --prompt "Is there a liver lesion?" \
  --grounded-prompt "Is there a liver lesion? Ground your answer with the mask." \
  --output outputs/holoct_8b_vqa

For target segmentation:

CUDA_VISIBLE_DEVICES=0 python -m inference.scripts.demo_seg \
  --model weights/HoloCT-8B \
  --ct demo_ct/my_ct.nii.gz \
  --prompt "Segment the liver in this CT scan." \
  --output outputs/holoct_8b_seg

The complete set of six demos covers lesion screening, VQA, RadGenome reporting, ReXGrounding reporting, target segmentation and finding-level referring segmentation. Each runs on one visible GPU with PyTorch SDPA.

Outputs and generation settings

Each run saves response.txt, response_raw.txt, result.json and a binary mask for each generated <SEG> token (mask_000.nii.gz, mask_001.nii.gz, ...). Paired demos keep the two responses in ungrounded/ and grounded/ subdirectories. Use a new output directory for each run.

All demos use 1024 maximum new tokens and seed 42. Screening, VQA and segmentation use greedy decoding. Report generation uses temperature 0.15, top-p 0.85 and repetition penalty 1.15. Masks are thresholded at 0.5 after inverse transformation.

Evaluation examples

The single-sample metric examples cover Dice, HD95, screening decisions, text overlap, clinical text scores and finding-level grounded-report quality. Each invocation takes one prediction and its corresponding reference.

Citation

If you use HoloCT, please cite our arXiv preprint:

@misc{xie2026holoct,
  title={A 3D foundation model unifying CT interpretation and segmentation},
  author={Xie, Yulu and Li, Zilong and Xu, Shen and Bai, Yifan and Shi, Liuhong and Chen, Jiao and Kong, Xianglei and Zhu, Wenchao and Shen, Na and Ye, Xianghua and Yan, Yang and Zheng, Jiaju and Peng, Yunsong and Su, Yanzhou and Yu, Chaohui and Luo, Hao and Wang, Fan},
  year={2026},
  eprint={2611.06471},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.06471}
}

Acknowledgements and license

HoloCT builds on Qwen3-VL, Med3DVLM and MONAI VISTA.

HoloCT-8B model weights are released under the MIT license. Component attributions are listed in the project's third-party notices.