One Operator, Many Robots: UAV–UGV teams for search and rescue

September 2026 · Submitted to IEEE SSRR 2026 (under review)

Imagine a rescue mission in a large, unstructured area: you need to cover ground fast, find anyone who might be lying there, and get help to them quickly. Now imagine doing all of that with a single person giving voice commands to a whole team of drones and one ground robot — no joysticks, no programming, no control room full of specialists.

That's the vision behind our latest work, One Operator, Many Robots: Natural-Language Command of UAV–UGV Teams for Search, Victim Detection, and Rescue, submitted to IEEE SSRR 2026. It builds directly on Swarm-Steward and takes it one crucial step further: connecting what the drones see with what the ground robot does.

Figure 1 shows the end-to-end workflow demonstrated in the field: (a) voice-driven mission control, (b) multi-UAV aerial search and victim localization, and (c) UGV deployment toward the confirmed victim location.

Figure 1 — End-to-end SAR workflow demonstrated in the field: (a) voice-driven mission control with *Swarm-Steward*, (b) multi-UAV aerial search and victim localization, and (c) UGV deployment toward the confirmed victim location.

From aerial search to ground response

Search-and-rescue operations require rapid area coverage, reliable victim localization, and fast deployment of response assets — but connecting aerial perception with ground-robot action remains difficult for non-expert operators. During autonomous lawnmower-style search, DJI Mini 4 Pro UAVs stream video and telemetry to a perception pipeline that applies a YOLO26l aerial person detector, tracks detections over time, and projects candidate victims to ground coordinates using UAV pose, gimbal state, camera geometry, and terrain elevation.

Figure 2 shows the full detection and object-fusion pipeline: video streams are sampled and processed by the custom YOLO26l detector, detections are geolocated using telemetry and camera geometry, accumulated into accepted per-UAV tracks, fused across UAVs by the Object Manager, and surfaced to the operator as candidate objects that can be confirmed, rejected, or allowed to expire.

Victim-detection and object-fusion pipeline from UAV video to confirmed candidates — Figure 2 — Victim-detection and object-fusion pipeline. UAV video streams are sampled and processed by the custom YOLO26l person detector. Detections are geolocated using telemetry and camera geometry, accumulated into accepted per-UAV tracks, fused across UAVs by the *Object Manager*, and surfaced to the operator as candidate objects that can be confirmed, rejected, or allowed to expire.

Accepted tracks are surfaced on the mission map as dynamic entities that the operator can confirm or reject — and once confirmed, they can be referenced in follow-up voice commands, for example to send a UAV to monitor the victim or to dispatch the UGV toward a nearby rescue position. Figure 3 shows the extended architecture: the UI layer supports natural-language commands, previews, and confirmations; the LLM-MAS layer coordinates agents and deterministic managers; the bridge layer performs protocol translation, video decoding, detection, and data batching; and the platform layer contains the UAV and UGV robots.

Extended Swarm-Steward architecture: UI, LLM-MAS, bridge and platform layers — Figure 3 — Extended *Swarm-Steward* architecture for aerial search and ground response. The UI layer supports natural-language commands, previews, and confirmations; the LLM-MAS layer coordinates agents and deterministic managers; the bridge layer performs protocol translation, video decoding, detection, and data batching; and the platform layer contains the UAV and UGV robots.

Training the aerial person detector

Because aerial person detection at these altitudes is different from ground-level detection, we trained a custom YOLO26l model on a merged set of four aerial-person datasets — SARD, HERIDAL, LaDD, and NOMAD — summarized in Table 1.

Dataset	Orig. imgs	Used	Val. imgs	Selection / preprocessing
SARD	1,183	1,000	200	Person-like classes collapsed to person; largest avg bbox area preferred
HERIDAL	1,546	~985	197	CSV labels converted to YOLO; largest avg bbox area preferred
LaDD	1,220	1,000	200	COCO labels converted to YOLO; largest avg bbox area preferred
NOMAD	20,752	1,000	200	Altitude-stratified 10/30/50 m sampling with round-robin actor selection

Table 1 — Training data used for the AerialPerson-Mix detector.

Four aerial-person datasets merged and relabeled to a common person class (~4,000 training images).

On a balanced validation split, YOLO26l reached mAP50 = 0.832 and mAP50–95 = 0.501, the best of the whole sweep while retaining stronger recall than the larger YOLO26x — see Table 2. Inference runs on the ground station (RTX 5090), so throughput was never the bottleneck in the field.

Model	P	R	mAP50	mAP50–95	FPS	Size
YOLO26n	0.803	0.713	0.774	0.412	523.18	5.1 MiB
YOLO26s	0.844	0.765	0.823	0.467	490.58	19.4 MiB
YOLO26m	0.856	0.766	0.819	0.487	254.90	42.0 MiB
YOLO26l (chosen)	0.847	0.779	0.832	0.501	223.25	50.5 MiB
YOLO26x	0.860	0.747	0.814	0.500	111.75	112.8 MiB

Table 2 — Balanced validation results and inference-only throughput for the YOLO26 model sweep. FPS was measured on the ground-station laptop with an NVIDIA RTX 5090.

YOLO26l selected: best mAP50–95 (0.501) while retaining stronger recall than YOLO26x and a much smaller checkpoint. Inference runs on the ground station (RTX 5090), not onboard the DJI UAVs.

Twelve outdoor experiments

We evaluated the system in 12 outdoor experiments over a 47,362 m² unstructured area, covering four victim placements, one-, two-, and four-UAV teams, different gimbal angles, and search speeds. Figure 4 shows the search area, the planned lawnmower sweeps, and drone-view examples of the four victim placements (a–d), with the full search area and sweep plan below.

Location A — Figure 4 — Search area, planned lawnmower sweeps, and drone-view examples of the four victim placements used in the real-world experiments. The area spans approximately 199 m × 238 m, corresponding to 47,362 m² (4.74 ha). One UAV flies all sweeps, two UAVs split the area into two groups, and four UAVs each take one sweep region (colored paths S1–4 denote different sweep regions). Locations A, B, and D produced successful localizations within 10 m; location C was the difficult placement where the available runs did not localize the victim within the success threshold.

Location B — Figure 4 — Search area, planned lawnmower sweeps, and drone-view examples of the four victim placements used in the real-world experiments. The area spans approximately 199 m × 238 m, corresponding to 47,362 m² (4.74 ha). One UAV flies all sweeps, two UAVs split the area into two groups, and four UAVs each take one sweep region (colored paths S1–4 denote different sweep regions). Locations A, B, and D produced successful localizations within 10 m; location C was the difficult placement where the available runs did not localize the victim within the success threshold.

Three placements produced successful detections with best localization errors of 1.21 m, 2.29 m, and 1.92 m — and the difficult failure case highlighted the influence of terrain, visibility, viewpoint, and GNSS stability. Table 3 reports the representative outcomes per scenario.

Scenario (Loc./N)	Speed (m/s)	Gimbal	Detector	Alt. (m)	Track det.	Mean err. (m)	Mean conf.
A / 1	2	−90°	mini4	35	18	1.21 ± 0.58	0.67 ± 0.18
A / 2	2	−90°	mini2	40	50	1.99 ± 0.78	0.52 ± 0.12
A / 4	2	−90°	mini3	45	15	2.41 ± 0.51	0.73 ± 0.11
B / 1	4	−45°	mini4	35	38	8.80 ± 1.46	0.74 ± 0.14
B / 2	4	−45°	mini2	40	10	7.09 ± 1.25	0.60 ± 0.14
B / 4	2	−90°	mini2	40	34	2.29 ± 0.29	0.67 ± 0.14
C / 4	4	−90°	—	—	—	—	—
C / 4	4	−45°	—	—	—	—	—
D / 1	4	−45°	mini1	35	17	9.33 ± 0.56	0.66 ± 0.13
D / 2	4	−45°	mini2	40	12	3.13 ± 0.76	0.65 ± 0.16
D / 4	4	−45°	mini3	45	9	1.92 ± 0.06	0.47 ± 0.14
D / 1	4	−90°	mini1	80	47	6.37 ± 2.85	0.66 ± 0.13

Table 3 — Representative search-experiment localization outcomes. Scenario is reported as Loc./N, where Loc. is the victim placement and N is the number of UAVs. Dashes indicate not-found cases where visual inspection confirmed that the detector did not observe the victim. The 80 m high-altitude run is listed last.

Locations A, B, D localized within 10 m; location C (difficult placement) was not localized. A / 1 and B / 1 at 35 m used mini4; D / 1 at 80 m used mini1.

From detection to rescue response

Once the operator confirms a candidate, the victim becomes a shared mission target that can be referenced in follow-up voice commands — for example, commanding a UAV to orbit the victim for continued monitoring while the UGV is dispatched to a position 5 m north to assist with evacuation without driving directly over the person. Figure 5 shows this post-detection response.

UGV Rover J8 driving toward a confirmed victim location — Figure 5 — Post-detection response after operator confirmation. After *mini2* detects the candidate *person_117* and the operator confirms it, the victim becomes a shared mission target that can be referenced in follow-up voice commands. In this example, *UAV 1* is commanded to orbit the victim for continued monitoring while *mini2* continues searching the area, and the UGV *Rover J8* is sent to a position 5 m north of the victim to assist with evacuation without driving directly over the person.

Feedback from a professional firefighter

A professional firefighter from Benalmádena (Málaga) joined the field demonstration and gave direct operational feedback, supporting the relevance of sector-based search, persistent no-go areas, human confirmation, adaptive UAV inspection, and fast interruption mechanisms for SAR-oriented multi-robot deployment.

One Operator, Many Robots: Natural-Language Command of UAV–UGV Teams for Search, Victim Detection, and Rescue
Jarabo-Peñas, Bravo-Arrabal, Rolland, Christensen · Submitted to IEEE SSRR 2026