The Illusion of Sight: Why Multimodal Models Stumble on Web Interfaces

The race to build strong artificial intelligence (AGI) is pushing the industry toward ever larger and more universal architectures. We have taught models to reason, write sophisticated code, imitate the human voice and generate photorealistic images. It seems that only one step remains before we get full-fledged digital agents that can do any job on a computer. Behind the polished presentations, however, lies a fundamental engineering barrier: modern multimodal models still cannot interact with user interfaces efficiently. What a human sees as an intuitive set of buttons and fields becomes, for an algorithm, a bloated stream of data that is too expensive to process and works very unreliably.

The root of the problem is the gap between how AI perceives information and how the modern internet is built. Today there are two main approaches to making a model "see" a website. The first is feeding it the source code: HTML markup, CSS styles and JavaScript logic. The second is visual analysis of screenshots, or even a sequence of frames resembling a slideshow. Both methods, despite being widespread, are crutches that expose the shortcomings of current architectures. Backend tasks can be automated through reinforcement learning in closed environments. The frontend, by contrast, remains a chaotic and shifting environment, where the slightest deviation in styling can completely paralyze even the most advanced agent.

A closer look at the technical details shows why reading code is a dead end for mass-market services. Modern page markup is loaded with superfluous tags, service scripts and endless nested elements that carry no meaning for the action at hand. A model has to spend its context window on thousands of lines of code just to find a single active link.

The visual approach, based on pixel analysis, suffers from a different problem: excess noise. To the human eye, a "Log in" button remains a button whether it is rectangular or oval, blue or green. We recognize its function instantly.

For a neural network, it is an array of millions of pixels, where every shade of gray in the background and every shadow under an element adds computational load without adding useful information. Training a model to build an "internal representation" of an interface through pure vision requires enormous volumes of data and compute, which makes such solutions prohibitively expensive for business.

The consequences of this inefficiency reach well beyond pure engineering. The current brittleness of models when working with interfaces means we cannot trust autonomous agents with critical real-time processes. If a small change in a site's design breaks the AI's logic, such systems remain nothing more than toys or narrow specialist tools that need constant human supervision. What is more, the high cost of each request, driven by the need to analyze "heavy" screenshots or huge volumes of the DOM tree, stands in the way of mass adoption of AI assistants. We are stuck in a paradigm where a model either tries to imitate human vision without understanding structure, or analyzes structure without understanding visual semantics.

Ultimately, the industry needs a fundamentally new approach to forming the internal representation of interface data. Instead of forcing neural networks to dig through a junk heap of HTML tags or analyze every noisy pixel on the screen, we need to build an intermediate layer: a native semantic protocol. It should be a compact but informative representation that conveys the essence of UI elements to the model and spares it unnecessary computation.

Only when we stop trying to teach AI to see a site with "eyes" or with "code", and instead give it a way to understand the functional architecture directly, will digital agents be able to move beyond perpetual prototypes and become genuinely useful tools in our hands. For now, multimodality remains only an expensive and crude imitation of vision, one that stumbles over an ordinary button on a web page.