EspanStereo: new dataset reveals LLM stereotypes in Spanish-speaking cultures

Researchers published a paper on arXiv on July 10, 2026 about EspanStereo — the first Spanish-language dataset for evaluating cultural stereotypes in large language models. The dataset was created using a new collaborative annotation framework: an LLM generates candidates for stereotypical statements, and native speakers from specific countries validate them.

Why English-only benchmarks don't work

Most stereotype datasets for LLMs focus on English-language contexts — and this is not by accident. Manual data annotation in less commonly used languages is expensive, and recruiting annotators from specific cultural groups is technically and logistically challenging. As a result, language models trained and evaluated primarily on English data transmit the perspective of the Western Anglophone segment to the entire world.

The problem is particularly acute for the Spanish-speaking world. For 500+ million native speakers — encompassing cultures from Mexico to Argentina, from Spain to Chile — systematic tools for evaluating AI models for cultural bias simply did not exist. Meanwhile, Spanish-speaking regions actively use language models, which means bias in these systems has real consequences for hundreds of millions of people.

How EspanStereo works

The authors proposed a cost-effective framework for collaborative annotation by humans and LLMs:

• LLM generates a list of stereotypical statements for a given culture
• Native speakers from target countries (in-culture annotators) validate, correct, and supplement the list
• The dataset includes both documented stereotypes from scientific literature and regional prejudices previously undescribed in NLP research
• Coverage: Spain and several Latin American states

This approach reduces costs compared to fully manual annotation while preserving cultural accuracy that automatic methods cannot provide. The key point is that native speakers capture subtle regional prejudices invisible to a model without specialized data. This includes stereotypes specific to individual countries and absent from existing English-language resources.

What tests of Spanish-language LLMs revealed

Evaluating modern Spanish-language language models using EspanStereo revealed significant differences in stereotypical behavior depending on the country. The same model demonstrates varying degrees of bias when it comes to Mexicans, Argentines, Spanish people, or Colombians.

"Our evaluation of

Spanish-language LLMs revealed significant variations in stereotypical behavior across countries, highlighting the need for more culturally grounded assessments," write the authors.

This confirms a systemic flaw in current methodology: benchmarks oriented toward American English miss cultural bias beyond the English-speaking world. Models may successfully pass standard stereotype tests — and simultaneously reproduce prejudices against Latin American cultures.

Scalability beyond Spanish

The authors particularly emphasize: the framework is not limited to Spanish. The collaborative annotation method adapts to any language and region — Arabic, Hindi, Swahili, Russian, Chinese. This lays the foundation for scalable multilingual stereotype benchmarks, the systematic creation of which was previously unprofitable due to high annotation costs.

If the approach gains traction, it could significantly expand the scope of LLM bias research — from narrow Anglophone focus to genuinely global, encompassing diverse languages and cultures.

What this means

EspanStereo is a concrete step toward fairer multilingual evaluation of language models. The collaborative annotation methodology lowers the entry barrier for teams working with culturally specific data. If the framework proves reproducible, similar datasets for other languages could emerge significantly faster. Until such tools exist, cultural bias in LLMs remains practically unmeasurable — especially for markets beyond the English-speaking world.