CloudTrack: Scalable UAV Tracking with Cloud Semantics

Authors: Yannik Blei, Michael Krawez, Nisarga Nilavadi, Tanja Katharina Kaiser, Wolfram Burgard
ICRA 2025

Nowadays, unmanned aerial vehicles (UAVs) are commonly used in search and rescue scenarios to gather information in the search area. The automatic identification of the person searched for in aerial footage could increase the autonomy of such systems, reduce the search time, and thus increase the missed person's chances of survival. In this paper, we present a novel approach to perform semantically conditioned open vocabulary object tracking that is specifically designed to cope with the limitations of UAV hardware. Our approach has several advantages. It can run with verbal descriptions of the missing person, e.g., the color of the shirt, it does not require dedicated training to execute the mission and can efficiently track a potentially moving person. Our experimental results demonstrate the versatility and efficacy of our approach.

Refer to the official publication and the video for more information.

Installation:

We recommend using python 3.10.12.

1. Clone the repository

git clone https://github.com/utn-blei/CloudTrack

2. Install the dependencies

cd CloudTrack
python -m venv ./.cloudtrack
source ./.cloudtrack/bin/activate
pip install wheel
pip install setuptools
pip install typeguard
pip install -e .[backend]
pip install --no-build-isolation git+https://github.com/IDEA-Research/GroundingDINO.git@856dde20aee659246248e20734ef9ba5214f5e44

3. Download the weights

mkdir -p models/groundingdino && cd models/groundingdino
wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
cd ../..

Run the Demo:

First: Run the backend

Run with LLaVA 13b as vlm (works out-of-the box):

python -m cloud_track backend

To use ChatGPT as a VLM, create an OpenAI API Key and call:

OPENAI_API_KEY=<Your Key Here> && python -m cloud_track backend --vlm gpt-4o-mini

Second: Run the Demos

The demo connects to the backend via network. We assume both instances run on the same machine. Otherwise, please change IP address and port accordingly.

Run demo on a sample video:

python -m cloud_track video-demo --video-source assets/faint.mp4 --ip http://0.0.0.0 --port 3000

Run the interactive demo:

python -m cloud_track live-demo --video-source 0 --ip http://0.0.0.0 --port 3000 --cathegory "a person" --description "The missing person is unconscious."

Use CloudTrack as a baseline:

We support easy inference on your personal data for benchmarking purposes. The following command supports input from a folder of images or a video file.

To run inference on your personal data, call:

python -m cloud_track video-demo --video-source 0 --ip http://0.0.0.0 --port 3000 --output-file "./output.json"

See minimal_example.py if you need more information for your analysis.

Known issues:

Freeze on Raspberry Pi (and potentially on other platforms):

OpenCV has a bug, where importing av leads to 100% cpu utilization when calling cv2.imshow(). (See here)

We found a workaround:

sudo apt-get install libavdevice-dev
pip install av==11.0.0 --no-binary av

If you use CloudTrack in academic work, please cite:

@inproceedings{Blei_2025,
     title={CloudTrack: Scalable UAV Tracking with Cloud Semantics},
     booktitle={2025 IEEE International Conference on Robotics and Automation (ICRA)},
     publisher={IEEE},
     author={Blei, Yannik and Krawez, Michael and Nilavadi, Nisarga and Kaiser, Tanja Katharina and Burgard, Wolfram},
     year={2025},
     month=may, pages={15893--15899},
     }