Josefine Birkmann

Author directory

2026

Cultural representation is a prerequisite for the comprehensive evaluation of cultural competence in AI systems. Yet there is no shared, structured way to assess what cultural representation in AI benchmarks involves or how it can be measured. We address this gap by introducing a six-dimensional framework that assesses cultural representation through a structured, dimension-specific scoring instrument, enabling systematic comparison across benchmarks. Applying our framework to 50 benchmarks explicitly targeting culture, we find that none achieves broad coverage across all dimensions. While procedural dimensions such as transparency and bias awareness are comparatively well addressed, benchmarks consistently score lower in linguistic and geographic representation, thematic breadth, and the conceptual grounding of culture. Strikingly, only 32% of benchmarks explicitly define culture, and annotator demographic information is absent in 82% of cases. By making these properties measurable and directly comparable, our framework provides a practical foundation for designing and evaluating benchmarks that more validly assess cultural competence in AI systems.