Researcher proposes four rungs of meaning for AI, from nouns to pacts

The author of the blog Tiled with Pentagons has posted a text on the AI Alignment Forum that he himself calls "a half-built conceptual tower with wet mortar." In its abridged form it runs about 12,000 words. The goal is to describe how two reasoning agents with different internal world models but similar predictions can understand each other on four levels of language, from the names of things to shared commitments. The author states plainly that he does not yet have the math needed for a real theorem. He considers some of his conclusions almost right, some merely suggestive, and many probably wrong. The main version of the talk was given in Berkeley, at the Lighthaven venue.

A ladder of four rungs

The starting point is John Wentworth's idea of interoperable semantics. If two Bayesian agents make the same predictions, they can agree at least on nouns as clusters in thingspace, even when they are built differently inside. What is new is that the author then stacks three more levels on top:

• Nouns: clusters in thingspace, the base level of agreement.
• Verbs: processes and changes that rely on nouns already agreed upon.
• Rules: constraints on whole sets of trajectories.
• Pacts: sets of rules that several agents take on at once.

Each rung has its own operators. A rung is characterized by which types the agents must already understand jointly in order to translate each other's messages. The central conjecture, for whose refutation or proof the author has set bounties, is that the mechanics of any rung do not work without the rungs beneath it.

Mauritius, X-Men and specification gaming

The author illustrates his motivation with a court case from 2003. In the US Court of International Trade, Marvel argued that the X-Men are not human. The law imposed a higher tariff on "dolls," meaning figures of people, and a lower one on other toys. To save on taxes, the corporation undermined the very frame on which its comics stand. The author puts it this way: when irresistible money met an immovable class of nouns, the class moved.

Then comes the key step. Replace the tariff schedule with a reward function and you get specification gaming: the adversarial redrawing of categories from above by those with stakes and leverage. The author promises to cover this in more detail in the next post. His historical witness is a different one: the School of Names in China, about 2300 years ago. Its adherents spent decades arguing over whether a white horse is a horse. Every school of the Warring States era laughed at them, and their scrolls were, according to tradition, burned under Qin Shi Huang.

What is testable here

The author points to an empirical trace. Where two rungs meet, the world's languages (or related cognitive technologies such as contract law) should contain a fuzzy class of words that bridges the gap between them. Among his reference points he names the work of Gärdenfors, Tarski and Gentner, as well as the later Mohists. He himself does not know whether to treat this proximity as a warning sign or as a moat that is unusually easy for him to cross. The text also mentions Borges's Tlön and worlds made of waves, where verbs appear before nouns.

The alignment part is expressed in a single thesis: the values we would like to pass on to AI can be expressed only as rules and pacts. At the same time, there is no theorem about when your concept of "murder" is a function of mine.

How to work with this

This is not a finished method but a research program with open questions. Two ideas are useful for practitioners who build multi-agent systems or write behavior specifications for models. First, failures in rule wording should be sought on the lower rungs: if agents have not agreed on nouns and verbs, a pact built on top of them will be brittle.

Second, specification gaming is naturally described as a revision of categories from above, not as a bug in a particular reward function. To those with a mathematical background, the author directly offers a challenge: break the construction and bring back a fix. Bounties are promised for this.

Before building tools on it, it is worth waiting for a formal version, which does not yet exist.