Evaluation is an essential process in the development and use of NLP systems. Benchmarks are widely used to calibrate systems, determine the efficiency of techniques, but also to advertise recently launched language models. Not only do benchmarks allow for evaluation and system comparisons, but they are also commonly utilized as training and validation corpora. In this paper, we aim at characterizing the "world-knowledge" that is put forth by benchmarks by closely inspecting their content. Through qualitative manual annotations (+15k instances) as well as quantitative automatic analyses, we uncover the presence of both explicit and implicit biases in MMLU, CommonSenseQA and MGSM, ranging from representational imbalance to bluntly sexist and racist instances. Our findings demonstrate that current NLP evaluation practices pose numerous quality issues, reinforce stereotypes and even contain explicitly harmful content. The current operationalization of benchmarks is questioned, and we provide recommendations and directions for further ethical evaluation. Warning: this paper contains content that may be offensive or upsetting.