Cosmos generated training video for drones: detector mAP@50–95 up 10.3%

A team from The Ohio State University tested whether generated video can replace some drone flights. The researchers took NVIDIA's open world model Cosmos-Predict1-7B Video2World, fed it nine frames each from real wildlife footage, and got back scene continuations of about 120 frames. A detector trained only on selected synthetic frames proved more accurate on real field videos than a detector trained on real data: mAP@50–95 (mean average precision at strict box-overlap thresholds) rose from 0.845 to 0.932, an increase of 10.3%.

How nine frames become a training set

The source dataset is MMLA: 37 low-altitude drone videos from Kenya and Ohio, with 155 074 annotated frames. For training, the authors used 20 618 frames containing 85 967 boxes around zebras, giraffes and onagers. Each Cosmos run received nine consecutive frames from the training split plus a short text description of the scene. The seed frames fix the camera, altitude, terrain and lighting, so the model invents only how the scene develops, not how it looks. This closes much of the sim-to-real gap, which classical simulators try to bridge with texture randomization.

The test videos were never used for seeding, prompting or selection. The only variable is the training data: the detector (YOLO11m on MMLA, YOLOv8m on VisDrone), the hyperparameters and the test set are all fixed.

The model makes frequent mistakes: animals sprout extra heads, bodies merge or vanish, and motion breaks physics. So an expert watched every clip, rejected any with a defect, and annotated the frames of the accepted ones. In a pilot of 40 clips, five passed, which gives the 87.5% reject estimate. The result was 26 702 synthetic frames with 93 004 boxes, which grew the training set 2.3 times in frame count.

What the measurements showed

On MMLA, adding synthetic data to real data raised mAP@50–95 from 0.845 to 0.881 (+4.3%), and the gain grew steadily with volume. mAP@50, however, stayed around 0.98: synthetic data does not help find more animals, it sharpens box boundaries. Training on synthetic data alone gave 0.932 and beat the real-data baseline in every class: zebra from 0.787 to 0.939, onager from 0.854 to 0.950, giraffe from 0.894 to 0.907. Even so, this model's mAP@50, precision and recall are not higher than those of the real-data model.

The authors openly note that single runs cannot separate the causes. The synthetic pool is 29.5% larger than the real one in frames. The generated frames probably contain less motion blur and sensor noise. And one expert annotated them in a single pass, whereas a large team produced the real labels. All three factors help strict localization in particular.

On the urban VisDrone benchmark (24 201 training frames, ten classes), mAP@50–95 rose from 0.138 to 0.184 (+33.5%) when 75% of the selected pool was added. Rare classes gained the most: truck from 0.050 to 0.277, van from 0.086 to 0.195. Of 300 clips, the expert accepted 76, and the 224 rejected clips (27 896 frames) went into a control group. They helped too, but the bus, the rarest class, collapsed from 0.211 to 0.010 in that model. Selection reaches the same average result with a quarter of the synthetic frames and protects rare classes.

What it cost

Generation ran on a single H100 at 330 seconds per clip, priced at $6.93 per GPU-hour. The full MMLA pool took about 163 hours and $1 131. Of that, 143 hours ($989) went to clips nobody used, so an accepted clip costs $5.08 rather than $0.64. On VisDrone, twice as many clips were accepted, but three quarters of the $191 also paid for rejects.

Real data costs more. The MMLA campaign took more than 150 person-hours across two continents. The training share (58%) is about 87 person-hours, or $3 210 at the median US biologist wage alone, not counting flights, lodging or flight permits. Synthetic data costs $1 131 plus about $164 for the expert's review of roughly 1 780 clips, making it 2.5 times cheaper. Without the failed generations, the gap would reach roughly 20 times: accepted clips took only $141 of GPU time. The authors excluded annotation from the calculation because both approaches need it.

The conclusions have limits. The reject rate was estimated from a small pilot, an expert is needed for every clip, and individual classes can quietly degrade behind a good average. The main direction for future work is stopping clearly bad generations early and replacing manual review with automated checks. If those work, rejects will no longer be the biggest cost item.