A marketer built an AI competitor monitor that spots unannounced features without flagging every deployment

The most interesting changes to competitors’ products often go unannounced. A new automation feature or AI assistant simply appears for users, without a press release or an app store entry. Vladimir Mikhailov, Marketing Analyst at Garage Eight, decided to catch these changes automatically. His experience is useful to anyone who has tried building a monitor with Playwright and drowned in false alarms.

Why DOM comparisons don’t work

The first idea is obvious: capture the HTML today, capture it tomorrow, and compare. On a modern SPA, that gives you a frontend deployment detector rather than a product change detector. A CSS class, testid, or section URL changes; a component moves; a button gets renamed; a user sees a different A/B test variant. All of this generates hundreds of alerts a day, almost all useless.

The author changed the question itself. Instead of “what changed on the page,” he asked “what can the user do now that they couldn’t do before?” That leads to the key architectural decision: the LLM does not receive two HTML files with a request to “find the difference.” The model works earlier, at the input stage: it explores the interface and builds a structured model of the product. Deterministic Python code compares the two models. The result is stable and reproducible, rather than dependent on the model’s mood.

An example from the article: a feature moves from /analytics to /insights/analytics, the DOM gets rewritten, and the user notices nothing. A structural diff will scream “MAJOR CHANGE!”. But if the URL and name stay the same while a manual action gains scheduled automatic execution, the diff stays silent, even though this is an important change. That is why each feature has a list of capabilities: schedule_action, export_report, create_custom_alert. Those capabilities get a separate hash, which identifies substantive changes.

The pipeline and its two interfaces

The sequence is: browser with an authenticated session, AI Discovery, feature model, semantic snapshot, comparison, business significance filter, Slack. The public website is just the storefront. Saved Views, Custom Alerts, Automation Rules, and recommendations are visible only after login, so the system navigates the product like an ordinary logged-in user.

An analytical layer sits on top. Claude connects through MCP to the monitoring history using read-only tools: competitors’ current features, NEW/UPDATED/REMOVED over a given period, category filters, and coverage checks. This provides two modes. Slack pushes changes automatically. And when an analyst asks “what have competitors added over the past month that the others don’t have?”, they get an answer in plain language, without SQL or JSON. The history also syncs to Google Drive as a knowledge base for a separate Claude Project.

Authentication: where automation breaks

Much of the article covers how to keep login from breaking the automation. Some practical findings:

• States instead of URL checks. Checking that “the URL does not contain /login” is unreliable: an SPA can briefly show one screen and immediately redirect to another. The author uses a state machine (AUTH_OK, AUTH_EXPIRED, AUTH_2FA_REQUIRED, AUTH_CAPTCHA_REQUIRED, AUTH_UNKNOWN) and considers the user authenticated when at least two of three independent markers are present, such as a user menu and a logout button.
• Navigation errors do not mean login failed. An “Execution context was destroyed” exception after a successful login can trigger another form submission. The system checks the state repeatedly and skips transient errors.
• Secrets never reach the model. The username and password live in the system credential store, and a separate standalone Playwright process enters them. The AI receives only a signal that the session is active again, and the model’s request history contains no secrets.
• 2FA is not bypassed. If an SMS arrives, the system stops and waits for a person. Once the code is entered, the session is saved, and subsequent runs proceed automatically.
• Not all CAPTCHAs are the same. An invisible reCAPTCHA badge on a page does not mean you are currently being asked to complete a challenge. The system needs to distinguish between “a CAPTCHA exists” and “a CAPTCHA is blocking login.”
• A persistent profile does not mean a persistent session. One service issued an auth cookie with expires: -1, which died with the Chrome process. The author chose to work with that behavior: after automatic login, the browser stays open, and Playwright MCP connects to it through CDP (connectOverCDP).

What to take away and what to keep in mind

Three ideas transfer well. First: use an LLM to turn chaos into structure, then compare it with deterministic code. Second: describe the product through user actions rather than markup. Third: treat secrets as an area the model never enters, by design.

There are limitations, too. The accessible portion of the material provides no figures on the cost of a run, how often the model misses or invents features, or how the significance filter distinguishes important changes from minor ones. Accessing other companies’ products through an account may violate a service’s terms of use, so that is worth checking with legal counsel before launch. The architecture itself nevertheless looks viable and suitable for product intelligence in any team.