IBI-001064 - Severepublie
Non-representative Benchmark Bias
Biais de benchmark non représentatif
IBI Score
7.2
out of 10
Severity4_grave
Frequencyfrequent
Scope groups4
Detection difficulty3
🇫🇷 Définition (FR)
Benchmark d'évaluation qui ne représente pas la diversité des cas d'usage réels.
🇬🇧 Definition (EN)
Evaluation benchmark that does not represent diversity of real-world use cases.
TriAxis Taxonomy
Structural Classification
O
Origin
O2
Computational & Statistical
L
Lifecycle
L6
Evaluation
S
Specific
S17
Measurement error bias
💡
Concrete Example
ImageNet contient des images biaisées qui ne reflètent pas la diversité visuelle mondiale.
📊 Detection Metrics
Demographic Parity DifferenceDisparate Impact RatioEqualized Odds Difference
⚙ Alert Thresholds
Disparate Impact Ratio0.8
Equalized Odds Difference0.1
Demographic Parity Difference0.1
📚 Sources & Scientific References
- •Do ImageNet Classifiers Generalize to ImageNet?Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar (2019)
- •Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI EvaluationMaria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, David Fernandez-Llorca (2025)
- •Identifying Statistical Bias in Dataset ReplicationLogan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Jacob Steinhardt, Aleksander Madry (2020)
🛡 Impact & Mitigation
🎯 Vulnerable groups impacted
No specific groups.
🛠 Mitigation methodologies
- AUDIT_TIERS : Third-party Audit
🏢 Sectors concerned
Media & Social Platforms (Risk: /5)Healthcare & Medicine (Risk: /5)
⚖ Applicable regulations
🌍 International
- •ISO_42001 : ISO/IEC 42001:2023
- •UNESCO_2021 : UNESCO - Recommandation sur l'éthique de l'IA
- •OECD_AI_PRINCIPLES : OCDE - Principes IA
- •COE_AI_CONVENTION_INT : Conseil de l'Europe - Convention IA
📋 Metadata
Contributor
Mohamadou SIDIBE
Institution
AI BIAIS INITIATIVE
Version
1
Created on
October 14, 2026
Updated on
November 4, 2026
UUID
8733e3c4-8d1f-4b6e-ba56-f3761418217b
Explore other biases
250 documented biases, classified by the TriAxis taxonomy.