FineART-VLA-subtask-midtrained

Predict the next subtask in language, then act on it.

FineART-VLA-subtask-midtrained is a vision-language-action checkpoint mid-trained on FineART, released by the FineART team at Scale AI and Hugging Face LeRobot. It combines language-based subtask prediction with continuous bimanual control in a single policy, building on Physical Intelligence's π₀.₅ using a native LeRobot conversion of the π₀.₅ initialization.

Paper · Dataset · Code · Usage guide

Camera views + task → next subtask in language
Camera views + subtask + robot state → continuous action chunk

Use this checkpoint to study language-guided ALOHA manipulation or initialize adaptation to new tasks and embodiments.

Checkpoint at a glance

These values describe the configuration shipped in this repository.

The dataset also records cam_low; this checkpoint uses the three views listed above. The weights, processor definitions, normalization statistics, and fitted action tokenizer are included together.

Training

The public FineART release contains 40,543 episodes, 1,718 hours, and 533,913 subtask annotations across 151 tasks on a Stationary Trossen ALOHA-style robot.

The saved processor recipe mixes 30% task-to-subtask prediction with 70% subtask-conditioned action training. Its objectives combine flow matching, text cross-entropy, and FAST action-token cross-entropy with weights 10, 1, and 1, respectively. Knowledge insulation prevents the continuous-action loss from updating the vision-language backbone through its keys and values.

The archived training configuration specifies a 300,000-step schedule, AdamW with a peak learning rate of 2.5e-5, 2,000 warmup steps, and cosine decay to 5e-6. Task sampling uses alpha=0.5; seven episodes are excluded in that recorded configuration. See train_config.json and the paper for the run settings and experimental protocols. The archived training configuration retains its historical initialization name; loading this checkpoint uses the weights included here.

Get started

Install a current checkout of LeRobot with FineART-VLA support:

git clone https://github.com/huggingface/lerobot.git
cd lerobot
uv sync --extra fineart_vla

Load the checkpoint on a CUDA machine:

from lerobot.policies.fineart_vla.modeling_fineart_vla import FineARTVLAPolicy

policy = FineARTVLAPolicy.from_pretrained("lerobot/FineART-VLA-subtask-midtrained")
policy.eval()

Run Python inside the installed environment, for example with uv run python. The weights occupy approximately 9.35 GB on disk; runtime memory also includes activations and caches. The text tokenizer is google/paligemma-3b-pt-224; its Hub repository may require accepting Google's terms and authenticating.

For robot inference, load the saved processor pipelines and normalization statistics alongside the policy. Match the camera keys and joint order in config.json. Follow the FineART-VLA usage guide for interactive subtask generation, manual language guidance, and rollout setup.

Fine-tune on your data

Prepare a LeRobot dataset with a subtask timeline in its language annotations, then initialize from this checkpoint:

uv run lerobot-train \
  --dataset.repo_id=YOUR_ORG/YOUR_LANGUAGE_ANNOTATED_DATASET \
  --policy.path=lerobot/FineART-VLA-subtask-midtrained \
  --policy.device=cuda \
  --policy.dtype=bfloat16 \
  --policy.gradient_checkpointing=true \
  --batch_size=8 \
  --steps=30000 \
  --output_dir=outputs/fineart_vla_finetune

Retain the checkpoint's fast_skip_tokens=128 when loading or fine-tuning; changing it changes the vocabulary positions learned by these weights. Adapt the observation/action features and normalization to your embodiment, and inspect annotation alignment before a longer training run.

Evaluation and limitations

The paper evaluates subtask supervision, knowledge insulation, language guidance, and transfer from ALOHA to YAM. Those experiments include multiple ablations and downstream fine-tuned checkpoints; their results should be read with the corresponding evaluation protocol. This repository provides the ALOHA checkpoint mid-trained with subtask supervision.

New hardware and environments require adaptation and validation. Subtask predictions can be incorrect, and this subtask-trained checkpoint does not establish general visual-question-answering or memory capabilities. Before physical rollout, verify camera calibration, joint ordering, action scales, and robot limits under supervised operation.

License and attribution

The checkpoint retains the Gemma license inherited through its PaliGemma/π₀.₅ initialization. See LICENSE, NOTICE, and the Gemma Terms of Use, including the incorporated use restrictions. The LeRobot implementation is available under Apache 2.0 in the code repository.

Citation

If you use FineART-VLA or the FineART dataset, please cite:

@article{choghari2026fineart,
  title   = {{FineART}: Fine-Grained Annotated Robotic Trajectory Dataset and Vision-Language-Action Model for Bimanual Manipulation},
  author  = {Choghari, Jade and Kooijmans, Pepijn and Agarwal, Mansi and Ciftci, Yusuf Umut and Doriwala, Aseem and Weaver, Catherine and Sivapurapu, Mouli and Yang, Kai and Wolf, Thomas and Lee, Jackson and Mannam, Pragna},
  journal = {arXiv preprint arXiv:2609.36416},
  year    = {2026},
  doi     = {10.48550/arXiv.2609.36416},
  url     = {https://arxiv.org/abs/2609.36416}
}