Developer releases an overlay that lets AI agents draw arrows and boxes on your screen

Developer franzenzenhofer has published a project on GitHub called big-arrow-on-the-screen. It is a tool that lets an AI agent draw a large arrow, box, or text label on top of the user's screen. The Hacker News post earned 233 points and 93 comments, so the topic clearly struck a nerve: agents lack a simple way to show rather than tell.

What problem the overlay solves

Today most AI assistants talk to the user in text. "Open the settings in the top right corner, find the third item in the list" is a sentence the user has to translate into actions on their own. If the interface is unfamiliar and the button names differ from what the model remembers, a mistake is almost guaranteed.

An arrow on the screen removes that translation step. The agent identifies the right spot and marks it visually, and the person sees at once where to look. The idea is simple, and judging by the Hacker News reaction, that is exactly why it resonated. The repository states exactly this function: large arrows, boxes, and text controlled by the agent.

A mode between hinting and autonomy

Agents that control a computer themselves click, type, and open tabs. But not every task requires full autonomy, and not every user is ready to hand it over. The overlay offers a third option: the agent shows, the human acts.

The practical scenarios are obvious:

• training on new software and employee onboarding;
• remote support, where the assistant explains steps right on the screen;
• intent confirmation: the agent highlights the button it is about to press and waits for approval;
• interface review and testing, when you need to point at a specific bug.

The last item is especially interesting. A visual marker before an action works as a cheap control mechanism: the user sees the agent's intent before it becomes a click.

Risks discussed around such tools

A program that draws over all windows requires certain system permissions, which means it widens the attack surface. If the agent is vulnerable to prompt injection, an attacker could theoretically make it point an arrow at "Confirm payment" or "Allow access." A user accustomed to trusting the hints will click. This is reasoning, not a scenario described in the repository, but the question is the same for any agent overlay.

Another issue is accuracy. An arrow pointing two centimeters to the left of the right button is worse than a text description. The news material does not make clear how reliably the project determines coordinates across different resolutions and monitors.

What is known so far

• Project: big-arrow-on-the-screen, author franzenzenhofer, published on GitHub.
• Features: arrows, boxes, and text over the screen, controlled by an AI agent.
• Reaction: 233 points and 93 comments on Hacker News.
• Date it appeared in the feed: October 9, 2026.
• Not specified in the source: license, supported operating systems, how it integrates with specific agents and models.

How to approach the project in practice

If you are building an agent with screen access, open the repository and check three things: how the overlay receives commands (protocol or API), which system permissions it needs, and whether you can restrict which elements the agent is allowed to highlight. For internal support and training tools, this is probably the fastest way to add visual feedback without full computer use.

And read the 93 comments on Hacker News. Discussions of projects like this usually surface real limitations, from multi-monitor setups to conflicts with full-screen applications. For now this is a small idea with a clear use case, and a check like this will show how mature it is.