AIM Intelligence benchmark finds AI safety gaps across 10 countries
AIM Intelligence says its XL-SafetyBench research, accepted to NeurIPS 2026, found that frontier AI models often miss country-specific risks and cultural taboos. The results underscore a gap between English-centric safety testing and how models behave in local legal and cultural settings.
Why it matters: - AIM Intelligence says current AI safety tests can miss how models behave in local legal systems, fraud patterns and cultural norms. - The study suggests a model that looks safe in English may still fail in country-specific settings. - The paper argues that refusal behavior and cultural sensitivity should be measured separately, not folded into one safety score.
What happened: - AIM Intelligence announced that its paper, "XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity," was accepted to the NeurIPS 2026 Evaluations & Datasets Track. - The benchmark tests large language models across 10 countries using 5,500 prompts written in local languages and grounded in local laws, institutions and cultures. - The research involved 17 researchers from 10 institutions, including Microsoft, Korea AI Safety Institute, KT, Coinbase and the Technical University of Munich. - The paper, dataset and code are available on arXiv, Hugging Face and GitHub under a CC-BY 4.0 license.
The details: - XL-SafetyBench covers the United States, France, Germany, Spain, South Korea, Japan, India, Indonesia, Türkiye and the UAE. - The benchmark splits evaluation into two parts: a 4,500-prompt jailbreak test for local risk and a 1,000-scenario cultural sensitivity test. - The jailbreak set uses prompts tied to local laws, institutions and fraud schemes to see whether a model gives harmful answers. - One example uses Korea’s jeonse lease system in a prompt framed as an official letter from a professor at a fictitious research institute. - The cultural benchmark hides a local taboo inside an ordinary request and checks whether the model notices it. - One example asks for a thank-you gift recommendation in France and checks whether the model suggests chrysanthemums, which are associated with death and mourning there. - Two native-speaker experts per country reviewed every test case, for 20 reviewers in total. - The team evaluated 37 models in total: 10 global frontier models and 27 locally developed models. - Across the 10 frontier models, localized attack scenarios drew harmful responses 44.8% of the time on average. - The paper says that figure reflects the benchmark scenarios, not a real-world breach rate. - Models performed best in the United States, where the attack success rate was 34.5%. - Attack success rates were higher in South Korea at 52.8% and in the UAE at 57.0%. - When asked directly, models recognized cultural taboos 83.2% of the time on average. - When the taboo was hidden in an everyday request, recognition fell to 49.9%. - All 10 models declined the hidden-taboo prompts, and recognition dropped below 40% for India and Türkiye. - Most locally developed models recognized hidden taboos less than 15% of the time, and three scored 0%. - Among those local models, lower attack success rates were linked with more irrelevant or incoherent answers, with a correlation coefficient of -0.81. - The paper describes that pattern as "comprehension-masked safety," meaning a low failure rate may reflect misunderstanding rather than correct refusal.
Between the lines: - The research starts from the premise that safety evaluation remains heavily centered on English, with non-English tests often created by translating English items. - The paper cites a survey finding that more than 90% of roughly 300 safety studies did not cover non-English settings. - The result is a benchmark that tries to capture risks tied to laws, fraud schemes and taboos that vary by country. - Dasol Choi, the principal researcher who led the study, said the team will keep working on evaluation standards that fit each country’s laws, institutions and culture.
What's next: - AIM Intelligence says XL-SafetyBench can help companies and institutions verify AI systems against local standards before deployment. - The company expects the benchmark to be used for both global services and country-specific AI products. - NeurIPS 2026 will be held in December in Sydney, Atlanta and Paris, where the paper will be presented alongside main-track work.
The bottom line: - XL-SafetyBench adds a country-specific test that exposes a major gap in how AI safety is measured: a model can look safer in English while still missing localized harm and cultural context.
Disclaimer: This article was produced by AGP Wire with the assistance of artificial intelligence based on original source content and has been refined to improve clarity, structure, and readability. This content is provided on an “as is” basis. While care has been taken in its preparation, it may contain inaccuracies or omissions, and readers should consult the original source and independently verify key information where appropriate. This content is for informational purposes only and does not constitute legal, financial, investment, or other professional advice.
Sign up for:
Seoul Daily Journal
The daily local news briefing you can trust. Every day. Subscribe now.
Check Your Email!
We sent a one-time activation link to: .
Confirm it's you by clicking the email link.
If the email is not in your inbox, check spam or try again.
Welcome back!
is already signed up. Check your inbox for updates.