Language → Verified Air–Ground Co-Simulation

AURORA A Natural Language–Driven Agentic Framework for Understanding, Reasoning, and Orchestrating Reliable Air–Ground Co-Simulation

Keshu Wu1 · Hao Zhang1 · Rui Gan2 · Xiangbo Gao1 · Xiaopeng Li2 · Zhengzhong Tu1 · Yang Zhou1,✉
1Texas A&M University  ·  2University of Wisconsin–Madison
✉ yangzhou295@tamu.edu · keshuw@tamu.edu
Mission vocabulary

The whole mission language fits in five pictures

The prompt picks one mission — hover, track, or navigate; the climb and landing come with it. A vocabulary that small gives every phase a predicate to enter, a monitor to check, and a parameter to repair.

Animated mission templates: the drone climbs on ascend, holds station on hover, follows a vehicle on track, flies a survey circuit on navigate, and settles on land.

One sentence in, a verified scenario out

Six prompts, compiled and executed. Each clip shows one run four ways, on one clock.

Police–bus pursuit

track
Drone tracks a bus from 40 meters and radios the ego when a police car chases the bus through an intersection.
Four-panel replay, police–bus pursuit: ground view, drone camera, 3-D scene, top-down map.

A trigger the prompt only implies, synthesized by the planner and then checked against when it fired.

Multi-agent tracking

track
A van drives ahead of the ego car and a police follows behind it while two background cars circulate; a drone tracks the van from 35 meters.
Four-panel replay, multi-agent tracking: ground view, drone camera, 3-D scene, top-down map.

Four coupled agents and an aerial follower — the case that presses on the 8.6 m/s flight ceiling.

Ascend, then track

track
A drone ascends to 25 meters and then follows a truck as it drives through the streets.
Four-panel replay, ascend then track: ground view, drone camera, 3-D scene, top-down map.

“Then” is a gate, not a hint: tracking cannot begin before the climb completes, and the monitor checks that order on the trace.

Cooperative occlusion warning

hover
A parked truck hides a pedestrian, who starts crossing; a hovering drone spots the pedestrian and warns the ego, which then brakes.
Four-panel replay, the cooperative occlusion warning: ground view, drone camera, 3-D scene, top-down map.

The truck hides the pedestrian for 77% of the crossing; the drone keeps 98% visibility. The warning lands 200 ms after detection; the ego stops 9.1 m short.

Survey patrol

navigate
In clear weather at midday, a drone flies a rectangular survey route at 30 meters altitude over the streets around the ego car.
Four-panel replay, a survey patrol: ground view, drone camera, 3-D scene, top-down map.

Waypoints that fall inside buildings are snapped to clear airspace before routing, so the route flies at the altitude asked for.

Night survey patrol

navigate
In wet roads at night, a drone flies a rectangular survey route at 40 meters altitude over the streets around the ego car.
Four-panel replay, a night survey patrol: ground view, drone camera, 3-D scene, top-down map.

Weather and time of day are requirements like any other — wet roads, night, 40 m — set from the prompt and scored one by one.

Highlights

Language-driven air–ground simulation, recast as verified compilation.

148
benchmark prompts, 603 annotated requirements
52 → 76%
verified scenarios, before → after repair
98.4%
requirement-level prompt fidelity
5.8 ms
worst-case cross-simulator drift

The framework

Eight stages, one rule for which of them the language model may touch.

The language model may propose what a scenario contains, but never what it measures.
Understanding

Typed parsing with rejection

Free text is decoded into a typed language, retrieval-augmented (RAG) over a knowledge base built by measuring the simulators, not by reading documentation about them.

Reasoning

Decided before anything runs

Where the scenario can exist, when its events fire, whether its requirements are satisfiable at all — each answered by construction, not sampling.

Orchestration

Run, monitor, repair what failed

Both simulators step in lockstep, temporal-logic monitors score the outcome, and a bounded repair ladder edits only the parameters that could have caused the violation.

The AURORA pipeline: retrieval-grounded parsing into the Air–Ground Scenario Graph, deterministic grounding, planning and verification, lockstep execution, and the bounded repair ladder.
The AURORA pipeline. A retrieval-grounded, schema-bound parse yields the typed Air–Ground Scenario Graph; deterministic modules ground, plan and verify it; CARLA and AirSim execute in lockstep; monitoring drives the bounded repair ladder.

The Air–Ground Scenario Graph

One object for every downstream stage to quantify over.

Grounded scenario graph for a cooperative occlusion warning.
(a) hover
Grounded scenario graph for tracking over a degraded communication link.
(b) track
Grounded scenario graph for a survey patrol.
(c) navigate

Grounded graphs from executed runs. Node color gives the type, edge style the relation, measures edges dotted. (a) A cooperative occlusion warning. (b) Tracking over a degraded link. (c) A survey patrol.

AURORA-Bench

148 prompts, 603 checkable requirements, probes on both sides of feasibility.

Composition of AURORA-Bench: 148 prompts in eleven capability categories drawing on 603 reference requirements of nine types.
Composition of AURORA-Bench. Left: 148 prompts in eleven capability categories draw on 603 reference requirements of nine types; line width gives the count, color the type. Right: per-channel shares and their normalized entropy.

Case studies

One prompt through every representation the pipeline produces: language, graph, plan, execution, measurement.

Aggregate metrics hide the difference between a scenario that satisfied its criteria and one that satisfied them for the wrong reason. Every execution therefore archives a full artifact set — specification, grounded graph, per-tick trace, frames, renderings, timelines — and every number traces back to it.

Case study: communication-degraded tracking, in five panels.
Communication-degraded tracking. (a, b) 3-D and top-down views. (c) Grounded graph with the ascent gate, proximity trigger and degraded link. (d) Task automaton and schedule, planned against realized firings. (e) Truck–ego separation, the send/deliver exchange, the online robustness margin.
Case study: police–bus pursuit with a synthesized trigger, in five panels.
Police–bus pursuit with a synthesized trigger. (a, b) The pursuit ring, drone over the bus's path. (c) Grounded graph. (d) Task automaton and schedule: the synthesized trigger fires at t = 25.4 s against t = 0.8 s as prompted. (e) Chase gap, ego speed, tracking error.

Citation

@misc{wu2026aurora,
  title  = {AURORA: A Natural Language--Driven Agentic Framework for
            Understanding, Reasoning, and Orchestrating Reliable
            Air--Ground Co-Simulation},
  author = {Wu, Keshu and Zhang, Hao and Gan, Rui and Gao, Xiangbo
            and Li, Xiaopeng and Tu, Zhengzhong and Zhou, Yang},
  year   = {2026},
  note   = {Manuscript in preparation}
}