OpenAI, Anthropic, and Google Fear One Question from Technical Directors

AI tool providers for developers — from OpenAI and Anthropic to Google and dozens of AI startups building code-writing agents — share a common interest in ensuring that vice presidents of engineering never ask one specific question. Not "how many employees use the tool?" and not "how much has generated code volume grown?"

Their real fear is the question: "What are the real business results?" The adoption of AI in software development is indeed explosive. GitHub Copilot reports millions of active users.

Cursor, Replit, Windsurf, and dozens of lesser-known tools are capturing technology team budgets worldwide. According to industry research, more than 70% of developers at large companies already use at least one AI tool for code writing. The entire market for AI assistants for developers, analysts estimate, will exceed $10 billion by 2027.

But beneath these impressive figures lies a serious problem. Most engineering leaders measure usage metrics, not outcomes. They know how many developers activated a license, how many lines of code were suggested by AI and accepted, what percentage of the team "actively uses" the product.

But they don't track whether feature time-to-market has decreased, whether production defects have declined, whether technical debt has shrunk, or whether actual development velocity increased after AI implementation. This is a costly blind spot that, incidentally, suits tool providers perfectly. The reason for this gap is clear: measuring usage is easy, while measuring real results is extremely difficult.

To assess AI's true impact on team productivity, you need baseline data from before implementation, proper control groups, robust definitions of what "productivity" means (which is itself a subject of heated debate in the engineering community), and time — at least several quarters. Tool providers are interested in license renewals and compelling marketing stories of success, not rigorous research that might reveal modest or even negative effects in various scenarios. Few independent studies on this topic yield mixed results.

GitHub experiments showed a 55% productivity boost for specific tasks like writing an HTTP server from scratch. Other research — including from METR and independent labs — recorded significantly more modest gains or warned that increased code-writing speed is often offset by growing time spent reviewing and debugging AI-generated code. Reality depends heavily on task type, team experience level, and how well the tool-use process is structured.

A separate issue is the new generation of AI agents. While early tools like Copilot functioned as advanced autocomplete, 2025-2026 agents claim to autonomously complete entire tasks: from code writing to creating PRs and passing part of CI/CD. This raises the stakes: if you can't measure ROI from a basic AI assistant, how will you evaluate results from a semi-autonomous agent?

The main conclusion is simple but inconvenient for the market: engineering leaders should stop measuring usage and start designing real experiments with measurable outcomes before signing six-figure contracts on AI tools. Define metrics in advance — cycle time, defect rate, PR review time, time to onboard new developers. Compare teams with the tool against those without, rather than simply tracking overall "satisfaction".

Tool providers won't teach this — they have no incentive to. But the ability to ask an uncomfortable question is what separates engineering leaders who genuinely improve their teams' work from those who simply buy the illusion of progress.