DroneCATS-Cosmos3-Edge-2B

The fine-tuned 2B drone model from DroneCATS. It takes one drone camera frame and a command, and returns one action in JSON.

Usage

Requires transformers>=5.18.0, torch, torchvision and accelerate.

pip install "transformers>=5.18.0" torch torchvision accelerate

import json, math, torch
from huggingface_hub import hf_hub_download
from PIL import Image
from transformers import AutoProcessor, Cosmos3EdgeForConditionalGeneration

repo = "naver-ai/DroneCATS-Cosmos3-Edge-2B"
model = Cosmos3EdgeForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)
SYSTEM = open(hf_hub_download(repo, "prompts/system_prompt.txt")).read()
USER = open(hf_hub_download(repo, "prompts/user_prompt.txt")).read()

# Drone frame, at most 640 px wide
image = Image.open(hf_hub_download(repo, "examples/tello_egg_mid.jpg")).convert("RGB")
if image.width > 640:
    image = image.resize((640, int(image.height * 640 / image.width)), Image.BOX)
w, h = image.size

# Camera intrinsics
hfov_deg = 56.4                                   # horizontal FOV of your camera (DJI Tello: 56.4)
k = 1 / (2 * math.tan(math.radians(hfov_deg) / 2))
fx = round(k * w, 2)
camera = json.dumps({"width": w, "height": h, "fx": fx, "fy": fx})

messages = [
    {"role": "system", "content": [{"type": "text",
        "text": SYSTEM.format(frame_w=w, frame_h=h, quarter_x=w // 4, quarter_y=h // 4)}]},
    {"role": "user", "content": [
        {"type": "text", "text": USER.format(instruction="go to the egg",
                                             history="(none yet — this is the first step)")},
        {"type": "image", "image": image, "camera_aux": camera},
    ]},
]
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False,
                                     enable_thinking=False, skip_reasoning=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
action = json.loads(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
print(action)   # {'action': 'go', 'point': [402, 656], 'depth_m': 3.2}

Use the prompt files as they are. The model was trained on exactly this wording.

Every image needs camera_aux, the JSON string with width, height, fx, fy built above. If it is missing, no error is raised, but the literal <|camera_aux|> placeholder stays in the prompt and the model gets an input it was not trained on.

Output

depth_m is in canonical units. Multiply it by k from the code above to get metres forward (Tello: 3.2 × 0.932 ≈ 3.0 m).

For the next step, send the new frame and pass the last actions (up to 5, oldest first) as history:

  step 1 of 2: ROTATE angle=-90deg  [done, 1.5s]
  step 2 of 2: GO x=402 y=656 depth=3.2m  [in progress]  <- latest

The model only proposes actions. It does not avoid obstacles, so use your own safety layer when flying a real drone.

Training data

Fine-tuned on MAmmoTH-VL and MAmmoTH-VL2 subsets (VisualWebInstruct, LLaVA-CoT-100k), plus drone-action data (DroneCATS-Pilot-54K) built from NYU Depth v2, SUN RGB-D and TartanAir v2 (BSD-3-Clause). See Appendix C of the paper for details.

License

OpenMDW-1.1. Fine-tuned from NVIDIA Cosmos3-Edge.

Citation

@article{park2026dronecats,
  title   = {Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents
             for Drone Control: Commanding, Approaching, Tracking and Searching},
  author  = {Park, Jaewoo and Lee, Minyoung and Seo, Sukmin and Yim, Moonbin and
             Yoon, Hyunwook and Ryu, Dohoon and Kim, Daehee and Song, Myungseo and
             Byun, Jihyuk and Chang, Seunggyu and Kil, Taeho and Kim, Jiseob and
             Lee, Bado and Kim, Geewook},
  journal = {arXiv preprint arXiv:2609.01404},
  year    = {2026}
}