ActiveFly-Bench

Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception

GitHub Code arXiv Paper Hugging Face Dataset

Abstract

Aerial Embodied Perception

ActiveFly-Bench is introduced as a benchmark for connecting cyberspace reasoning and physical-world interaction in UAV embodied perception. It decomposes active perception into Aerial Embodied Question Answering, Observation Behavior Planning, and Fine-grained Language-guided UAV Control, explicitly linking high-level task understanding, behavior planning, and low-level control.

The datasets are collected from real-world and simulated outdoor environments. ActiveFly is further developed as a closed-loop UAV agent that integrates visual-language reasoning with fine-grained control. Experiments with representative VLMs and VLA models show that current agents still struggle with observation planning, viewpoint adjustment, and robust task completion.

Overview

Bridging Reasoning, Planning, and Aerial Control

ActiveFly-Bench is built for language-guided UAV active perception, where an agent must reason from a question, plan how to observe, execute fine-grained control, and answer from the gathered visual evidence.

01

First Unified UAV Benchmark

It bridges high-level embodied question answering and low-level UAV action execution instead of evaluating them as isolated capabilities.

02

Three Aligned Subtasks

Air-EQA, OBP, and FLUC share the same semantic route from question to observation behavior to executable UAV movement.

03

Real-World Agent

ActiveFly combines VLM planning with VLA control and is validated on a physical UAV through a ground-drone collaborative system.

Overview of ActiveFly-Bench
Figure 1. Overview of ActiveFly-Bench. The benchmark targets language-guided embodied perception for UAVs. It decomposes the task into three hierarchical and semantically aligned subtasks: Aerial Embodied Question Answering, Observation Behavior Planning, and Fine-Grained Language-Guided UAV Control. Given a high-level task such as “What is under the tree?”, the UAV first performs reasoning and planning to infer the observation behavior, then executes fine-grained control actions to reach an appropriate viewpoint, and finally answers the question based on the visual information.

Tasks

Three Semantically Aligned Subtasks

The benchmark asks whether a UAV can answer an embodied question by deciding where to look and then executing the motion needed to obtain the right viewpoint.

Air-EQA

Answer an aerial embodied question after actively collecting relevant observations.

OBP

Infer the observation behavior needed to answer the question from the initial view.

FLUC

Generate fine-grained UAV-body and gimbal actions that reach a task-relevant viewpoint.

Examples of Air-EQA, OBP, and FLUC tasks
Figure 2. Examples of the three tasks in ActiveFly-Bench. Air-EQA spans five question categories: object recognition, attribute recognition, counting, spatial understanding, and world knowledge, with each example illustrated by the first and last frames. Observation Behavior Planning bridges Air-EQA and FLUC. FLUC includes 10 elementary and 4 complex active perception trajectories.

Dataset

Real and Simulated UAV Active Perception Data

The benchmark is built from expert trajectories and human-validated QA/planning annotations, keeping the three subtasks aligned at the semantic level.

500 Active perception tasks
10K FLUC trajectories
2.6K QA and planning pairs
6.2K Simulation trajectories
1.9K Real indoor trajectories
1.9K Real outdoor trajectories
Dataset generation pipeline and statistics
Figure 3. Dataset Generation Pipeline and Statistics. (a) The data generation pipeline of three tasks. (b) The distribution of FLUC trajectory lengths. (c) Word-count distributions of language annotations. (d) Dataset composition statistics. (e) A word cloud generated from the language annotations. smooth trajectory, which is replayed by the UAV to automatically generate 5 to 40 demonstrations for each instruction.

Demo

Benchmark Demo Videos

The paired benchmark videos compare OpenVLA and Pi-0.5 on matched rollout cases, while the real-world collection demos are shown as standalone UAV videos.

OpenVLA Demo 1
Pi-0.5 Demo 1
OpenVLA Demo 2
Pi-0.5 Demo 2
OpenVLA Demo 3
Pi-0.5 Demo 3
OpenVLA Demo 4
Pi-0.5 Demo 4

Real-World Demos

Real Scene Collection

These demos are collected from real indoor and outdoor scenes, with two indoor cases and two outdoor cases shown as standalone examples.

Indoor Demo example 1
Indoor Demo example 2
Outdoor Demo example 1
Outdoor Demo example 2

Sample

Frame-Aligned Video and Motion Trajectory

A sample trajectory from the benchmark is shown with its first-person observation video and synchronized UAV motion path. The image sequence in sample/ is encoded at 0.15 seconds per frame.

Observation video generated from 80 sample frames at 0.15 s/frame.
Instruction

Move right until you are in front of the blue overhead billboard while adjusting camera left to keep the blue overhead billboard in the centre of the camera. Then move down until you are in front of the blue overhead billboard while adjusting camera up to keep the blue overhead billboard in the centre of the camera.

White-background 3D motion trajectory generated from sample/log.json, synchronized to the image frames.
right x24 turn_left x6 down x10 camera_up x5

ActiveFly Agent

Closed-Loop VLM + VLA System

ActiveFly first uses a VLM to plan the observation behavior, passes that plan to a VLA model for fine-grained UAV action prediction, and then answers the embodied question from sampled observation history.

ActiveFly
Question + Initial View
VLM Observation Plan
VLA UAV Actions
Answer from History
Plan

Observation Behavior

The VLM predicts how the UAV should move or adjust viewpoint to acquire missing information.

Act

Fine-Grained Control

The VLA consumes the observation plan, current observation, and UAV state to predict control actions.

Answer

History-Aware QA

After execution, the VLM answers from a sampled observation history; the paper uses 16 images in practice.