Cohere Labs Tested AI With Riddles: 100+ Authors From 50 Countries Found Cultural Gaps

Cohere Labs has run an open research project that exposed a weak spot in language models: they speak fluently, but know little about the cultures of the people who speak the language. More than 100 participants from over 50 countries wrote and evaluated riddles built around local realities, then watched how models trained mostly on English data handled them. The answers looked convincing but were often wrong. In Buginese, a language spoken by millions of people, the models showed practically no knowledge at all.

How Hungarian riddles became a global benchmark

It all started with Karoly Bochka, a Hungarian diplomat with an interest in AI research. He devised difficult quiz-style riddles in Hungarian and checked whether models picked up on the cultural clues. The result surprised him: the answers sounded right but didn't match reality. The models lacked concrete facts, and in places they flat-out hallucinated.

Bochka shared his observations with Julia Kreutzer, a Staff Research Scientist at Cohere Labs, during a discussion about culture in AI within the lab's open research community. Kreutzer saw a way to measure the gap between a model's fluency and its real cultural knowledge. Together they scaled up the project with a call on social media and in Cohere Labs' existing research communities, with no selection criteria. It was enough to belong to a culture and be ready to write and evaluate riddles.

Participants gathered in Discord and built benchmarks on their own, on topics such as sports, history and politics. Every question was framed as a riddle. The format was chosen deliberately: a riddle relies on ambiguity but has a concrete solution, so it tests both interpretation and factual knowledge.

A high school student and a teacher as researchers

The project's most prominent participants don't look like researchers on paper. Henry Gagnier, a 16-year-old American, joined the team and volunteered to lead one of the workstreams, though he admits he was timid at first. In his words, a benchmark like this requires not AI skills but real knowledge of a culture. The project gave him a sense of what researchers do, and he now builds his own projects to go deeper into the subject.

Jane Smith, a retired Canadian teacher, brought a familiar teaching technique to evaluating answers: working out exactly where the line of reasoning took a wrong turn and what led to the incorrect answer. For her riddles, she used imagery such as tigers and crabs to make the logic playful.

What the team documented:

• more than 100 participants from 50+ countries, with no expertise requirements to join;
• the community and benchmarks were organized in Discord;
• question topics: sports, history, politics and other cultural areas;
• models with an English skew in their training give plausible but wrong answers to local riddles;
• in Buginese, the models showed practically zero knowledge, and therefore zero knowledge of the culture tied to it.

A confident tone instead of "I don't know"

Gagnier was surprised that the problem went beyond a lack of cultural knowledge: the models didn't know the language itself. Smith noticed something else: the models very rarely answered "I don't know." Instead, they produced authoritative-sounding phrasing that makes it easy to believe the system understands the subject. In her words, that is exactly what is frightening, because people look for authoritative answers. Her conclusion: AI is not dangerous in itself; what is dangerous is a lack of critical thinking in the people who use it.

For the industry, this is a familiar but poorly solved calibration problem: a model should understand the limits of its knowledge and say so. Cohere Labs says it is working on better understanding of context and is studying how confident current models are. A related post on the lab's blog, published on October 20, covers the same topic, a rethink of approaches to confidence in reasoning models.

What to check before launching AI in a local market

If you are taking a product to a market with a non-English language or regional specifics, don't rely on general benchmarks. Put together a small test set with questions from native speakers: local organizations, laws, sporting events, idioms. Separately, measure the share of answers where the model honestly admits it doesn't know, not just the share of correct ones. For low-resource languages, build in manual review and limits on autonomous decisions. The Cohere Labs project shows that communities can assemble such test sets themselves, and that the best authors of these checks are people who simply know how things work where they live.