Why Multimodal Models Can't See Web Interfaces

Two Failed Strategies

Today, multimodal models interact with websites in two ways. The first is reading source code: HTML, CSS, JavaScript, backend. The second is looking at screenshots. In practice, both approaches are inefficient.

Code doesn't reveal reality. From the source markup, a model can't tell where elements actually end up after rendering, whether they overlap each other, or whether a button has drifted off-screen. Code doesn't show that an element exists in the markup but is inaccessible to the user. You can expand the context to a million tokens, but code still won't give you the interface in its real, rendered state.

Screenshots solve part of the problem, but at the cost of massive overhead. A single frame isn't enough. You need different viewport sizes, different element states, hover, focus, modal windows, loading processes, dynamic changes. Full understanding may require video. A task the browser already understands as element structure, geometry, and state turns into heavy pixel analysis — expensive and inefficient.

How Humans See

We look at a website differently. We almost instantly distinguish text, backgrounds, buttons, borders, input fields, menus, and modal windows. On hover, we see interactive changes: a button's color shifts, feedback appears. We don't need to reconstruct the interface from either code or pixels.

We see structure. We see an oval with the text "log in" and immediately understand it's a button. We see block boundaries and the hierarchy of elements. None of this requires analyzing every pixel or studying all the styles and markup.

A Third Layer of Perception

Multimodal models need to see the same way. Interfaces shouldn't be just code or an image to them — they should be a structured representation suited for understanding a page, checking layout, and interacting with the UI.

There's no need to abandon transformers. Models need a new built-in perception module — a way to see an interface as a structure of elements, their states, visibility, and geometry. Not as a temporary "hand" that an agent rewrites as a script each time, but as a fundamental mode of perception.

Such a module would enable solving a broad range of tasks — from interface analysis to development, from layout verification to automation — while remaining universal. This isn't an attempt to sell a ready-made solution, but an attempt to name the problem: multimodal models have a crude tool for working with interfaces and need a more specialized way of perceiving them.