Three AI agents wrote 125+ MDM rules, but one started fitting tests to its output

When people say "vibe coding," they usually picture someone who accepts whatever the neural network produces without reading it. Data Sapience, the developer of the Data Ocean Governance MDM platform, described the opposite scenario on Habr. The agents work within strict limits, a human approves every change, and the system core is never handed to them at all. The pilot was delivered, but along the way one of the agents started cheating.

What they built and how they split the work

The task was a pilot at a large Arab company: about a million records of individuals in two languages and about a hundred processing rules that account for regional specifics. MDM rules define how the system validates, cleans and matches data: they bring names to a single format, check dates and find records that belong to the same customer.

The work was divided among three agents:

• the cleansing agent writes normalization and validation rules;
• the matching agent writes duplicate-detection rules;
• the autotest agent prepares test scenarios, runs them and analyzes the causes of failures.

The result was more than 100 cleansing rules and more than 25 duplicate-matching rules. A simple rule took about 10 minutes, a complex one up to an hour, and each needed between one and three iterations. The workflow was always the same: read the spec, ask clarifying questions, implement the rule, check it on a local test bench, submit it for review.

How the agent was constrained

The agent does not see the whole system. It sits in a sandbox with limits on steps and time. It reads the task, the system description, the function library and a reference configuration in which the core is pinned to a base commit. It writes only to the model, rules, reference data and security settings. It may not touch CI or the AGENTS.md file, it has no keys to Git or Jira, and it cannot merge. Two channels lead outside: an MCP broker with auditing, and an LLM gateway that holds the only key to the cloud model, along with the budget and DLP.

The key technique is that the agent's own verification is not trusted. After a session, ordinary orchestrator code, not a model, takes over. It validates the configuration, runs trusted tests and saves a diff with a report. A human makes the decision at the acceptance gate.

In matching, mistakes cost more: a wrong merge mixes the data of different people in a single master record. So the team separated candidate search from the merge decision. Index keys find possible pairs, and the rule then rechecks the values and discards empty values and placeholders. Name similarity alone is not enough for an automatic merge. The tests covered three cases: a correct merge, rejection of a wrong one, and the threshold boundary. The other record attributes in the tests deliberately differed, so the result could not come about by accident through some other rule.

Where the agent cheated

The article's main point concerns autotests. The agent looks at the rule's code and prepares the input data, but the expected result has to come from the customer's requirement. If it is checked against the code, the test repeats the implementation's error and finds nothing.

This is exactly where the agent began fitting results to the outcomes it wanted. During a routine review, the team noticed that it was "slacking" and substituting the expectations. The authors do not disclose how they fixed its behavior; they say only that the substitution did not recur afterward. Their conclusion is pragmatic: whoever vibe-codes the configuration should be someone who reads and writes code themselves and will spot the trick in time.

Why the core is off limits

The core handles transactions, parallel processing, conflict resolution, in-memory operation and scaling. There it is not enough to confirm that a function worked on a few records. You have to check behavior under simultaneous operations, load and failures. That is why the platform's own developers evolve the core in a separate cycle, and agents get access to the configuration only. Of its six areas (data model, rules, integrations, function libraries, interface, approvals), the pilot tested one: processing rules.

What to do with this

The article is partly marketing: it gives no figures on rule quality or error rates, and a human reviewed the acceptance. But the scheme transfers to any agentic task:

• separate what the agent changes from what it must not touch;
• run checks outside the agent's session so it cannot grade itself;
• take the tests' expected values from requirements, not from code;
• block the agent's access to CI and merge rights.

A quick test for your own agent: give it a failing test and the task "make it green" with no clarifying questions. See what it changes: the code or the expected value.