How language models help train construction robots without manual data labeling

The construction industry remains one of the world's least automated sectors. Excavators, bulldozers, and cranes are still operated by humans, and labor productivity in construction has barely grown over the past decades—unlike industrial manufacturing, where robotics long ago became the norm. One of the main reasons for this gap is a catastrophic shortage of quality data for training autonomous systems. And it's precisely this problem that startup Bedrock Robotics has taken on, joining forces with Amazon Web Services.

The company joined the AWS Physical AI Fellowship program and gained access to resources from AWS Generative AI Innovation Center—an Amazon division that helps partners implement generative AI into real products. The task Bedrock Robotics set for itself sounds deceptively simple: teach construction equipment to work autonomously. But behind this formulation lies a fundamental problem of scaling data.

For an autonomous excavator to safely dig a trench or move earth, its neural network models must be trained on enormous volumes of annotated data. Every frame of video from a construction site must be annotated—marking the position of equipment, people, obstacles, determining the type of operation being performed, recording environmental context. Traditionally, this is done by annotation teams, and the process is expensive, slow, and doesn't scale well. For the construction industry, where every site is unique and conditions change hourly, this problem is especially acute.

Bedrock Robotics' solution relies on vision-language models—a class of multimodal systems capable of simultaneously "seeing" an image and "understanding" textual descriptions. These models analyze videos of construction work, automatically extract operational details from them, and generate annotated training datasets without human involvement. Essentially, instead of hiring hundreds of annotators, the startup delegates annotation to another neural network—and does so at scales inaccessible to manual labor.

Technically, the approach works as follows. A video stream from a construction site is fed into a vision-language model deployed on Amazon Bedrock infrastructure. The model analyzes what's happening frame by frame, recognizes types of equipment and operations being performed, determines spatial relationships between objects, and generates structured annotations. These annotations are then used as training data for specialized models that directly control autonomous equipment. It amounts to a kind of pipeline: a large universal model prepares data for small specialized models.

It's important to understand the context in which this solution has emerged. Physical AI—robots, autonomous vehicles, industrial manipulators—is experiencing a moment similar to what language models went through several years ago. The algorithms are already powerful enough, computational resources are available, but data remains the main constraint. Unlike text data, which can be gathered from the internet, or even images, of which there are billions online, data about physical operations is a rare and expensive resource. Every hour of video from a construction site needs not just to be recorded, but meaningfully annotated with domain-specific considerations.

Bedrock Robotics' approach potentially transforms the economics of the entire autonomous equipment industry. If data annotation ceases to be a bottleneck, companies can iterate their models much faster, train them on more diverse scenarios, and bring products to market more quickly. This applies not only to construction—similar logic is applicable to mining, agriculture, warehouse logistics, and any other field where physical systems must act autonomously in unstructured environments.

There are, however, questions to be raised. The quality of automatic annotation inevitably falls short of expert manual annotation, and errors in training data can cascade through to the final control models. For systems working alongside people on construction sites, the cost of error is measured not in pixels but in human lives. How reliable automatic annotation is in safety-critical scenarios—this is a question that doesn't yet have a public answer.

Nevertheless, the direction is set. Using generative AI to prepare data that trains another AI is not just an engineering trick, but a pattern taking shape across the entire industry. Amazon is clearly making a strategic bet on physical AI as the next major market after language models, and the Physical AI Fellowship program is part of that bet. Construction equipment that thinks for itself remains a matter of the future. But the data for that future is already beginning to be prepared by machines.