ProjDevBench: Can AI Build Complete Software from Scratch?

# ProjDevBench: Can AI Build Complete Software from Scratch?

When we talk about artificial intelligence in software development, we usually think of examples like ChatGPT fixing a bug in a function in a minute, or Claude generating neat code for a simple algorithm. But what happens if you ask an AI agent to design and build a complete application from scratch, with the full architecture, dependency management and component integration? Researchers at leading labs have quietly been working on this question and built ProjDevBench, a platform that exposes the real capabilities and limits of today's AI models acting as full software engineers. The results force a rethink of optimistic forecasts that automation will soon displace developers.

ProjDevBench is radically different from earlier tests of AI coding. Previously, researchers checked whether a model could write a single function or solve a LeetCode problem. The new benchmark sets AI a real task: build a finished product from scratch. The platform requires agents not just to generate code but to make architectural decisions, split a project into modules, manage dependencies, write tests and combine everything into a working product. These are not isolated functions. It simulates real development, where every decision affects the next and errors accumulate, complicating the whole system.

The structure of ProjDevBench itself mirrors the real challenges of software engineering. Agents receive project specifications of varying complexity, from simple utilities to applications with several layers of logic, databases and external APIs. Models must understand the requirements, plan the code structure, choose suitable technologies and libraries, resolve conflicts between components and make sure the result works. It is quite similar to what a junior developer does on their first serious assignment, except without the chance to ask a senior for advice and without pull request review.

The test results surprised even the optimists. Current LLM agents, powered by top models such as GPT-4 and Claude, do show progress over the previous generation. They can sensibly split a project into modules, pick a reasonable architecture and write functional code. But problems appear right away. Agents forget about dependencies between components and generate code that works in isolation but breaks at integration. They manage system state poorly and often cannot trace how a change in one module affects others. Code scalability degrades as complexity grows: agents start duplicating logic instead of refactoring, turning a simple project into a tangled mess.

What's more, AI developers handle project-level debugging poorly. When something goes wrong, models often lose the thread of cause and effect and start changing random parts of the code instead of analyzing the problem logically. Testing, which should be an integral part of development, often becomes a formality: agents write tests that pass because, in essence, they test the same thing the actual code does.

These results do not mean AI is useless for development. They show a real gap between generating code and engineering software. The first is a multiplication table; the second is an art. ProjDevBench makes clear that a fully autonomous AI developer is still a long way off. The future most likely belongs to hybrid tools: AI assistants that generate code and suggest solutions, but under the supervision of an experienced engineer who can think strategically and see the whole picture.