Conversation Context Changes AI-Generated CRM Shortlists by More Than Half

Ask an AI system which CRM a Russian company should choose, and you will get a list of familiar products. Precede the same question with a couple of sentences about the size of the sales department or a long sales cycle, and the list will change noticeably. Name a specific brand in advance, and it will very likely appear in the final answer as well.

This seems obvious until companies begin reducing their “AI visibility” to a single number. At that point, unprompted answers and answers produced after discussing a brand often end up in the same report. A pilot led by Pragmatix Digital head Andrey Kruzhkov shows why mixing them distorts the result.

One Question, Five Different Lead-Ins

The experiment involved ChatGPT, Perplexity, and Alice AI. Each system was asked the same question: which CRM platforms it would consider for a company in Russia. Only the preceding conversation changed.

In the control scenario, the question was asked immediately. In the other chats, it was preceded by a description of a small B2B company, a long sales cycle, a mention of BPMSoft, or an initial discussion of CRM products proposed by the AI itself. Each scenario was repeated three times for every system, producing 45 conversations in total.

To compare the lists, the author used the Jaccard coefficient: the share of products common to both answers among all unique CRM products in those answers. If the lists match completely, the result is 100%.

Without a lead-in, answers from the same system were relatively consistent: similarity across the three control selections was 83% for ChatGPT, 91% for Perplexity, and 67% for Alice. After a company description, however, the composition changed more substantially. In the small-team scenario, similarity to the control fell to 48% for ChatGPT, 23% for Perplexity, and 51% for Alice. Describing a long sales cycle produced 62%, 59%, and 46%, respectively.

Consider a simple example: the user did not alter the CRM question itself, but had previously said that the sales team had eight people and that a complex implementation was undesirable. That was enough for ChatGPT and Perplexity to restructure their selections more dramatically than the normal variation among repeated answers.

Mentioning a Brand Changes What the Metric Means

The clearest result emerged in the BPMSoft scenario. The user first said that a colleague had mentioned the product, then asked the AI to examine the broader market, and only afterward posed the general CRM question.

BPMSoft appeared in all nine final selections: three answers from each AI system. In all three ChatGPT cases, the product was listed first, although Bitrix24 held that position in the control answers. BPMSoft’s frequency increased from two appearances to three with ChatGPT and from one to three with Alice; it remained unchanged with Perplexity.

But this cannot be treated as evidence of strong independent brand visibility. The name was already present in the context, and in one conversation ChatGPT had time to explain what BPMSoft was and discuss the BPM and low-code market. The experiment measured the response to the entire conversation, not merely to the final question.

This creates an important distinction. The first metric answers whether the model will name a product without prompting when given a general query. The second shows whether it will retain an already mentioned product in its final recommendation. These are different tasks, and combining them into a single indicator is risky.

How to Run Tests Like These

For a useful GEO measurement, the scenario, model, search mode, wording, labeling rules, and number of repetitions must be fixed in advance. The entire conversation should be retained because intermediate answers also influence subsequent output. A fresh chat with no history and a conversation in which the brand has already been named should be tested separately.

The pilot has limitations. Each scenario had only three runs, all completed on a single day. Day-to-day variation was not tested, and the effects of the requirements themselves and the way they were presented were not separated. In one ChatGPT run, even the web-search mode differed. The figures therefore show the direction of the effect, not its stable magnitude.

The practical conclusion for product and marketing teams is simple: context is part of the test, not background noise. If it is not stored alongside the answer, a brand-visibility report becomes a mixture of independent recommendations, user cues, and random model variation.