The emergence of open data portals necessitates more attention to protecting sensitive data before datasets get published and used. To do so effectively, we observe the need to refine and broaden our definitions of sensitive data. We argue that the sensitivity of data depends on its context, and that this context should be considered in methods for assessing data sensitivity. To this end, we introduce a framework, termed ConSenser, for contextual sensitive data detection based on two aspects: 1) type contextualization, which considers the type of data values within the overall context of the dataset or document to assess their true sensitivity, and 2) domain contextualization, which assesses the sensitivity of data values informed by domain-specific information external to the dataset, such as geographic origin of a dataset. Experiments instrumented with language models confirm that type-contextualization significantly reduces the number of false positives for type-based sensitive data detection and reaches a recall of 94% compared to 63% with commonly used tools. Moreover, domain-contextualization through retrieval of sensitivity rules effectively grounds sensitive data detection in relevant context in non-standard data domains. A case study with humanitarian data experts also illustrates that context-grounded explanations provide useful guidance in manual data auditing processes. We open-source the code and annotated datasets at https://github.com/trl-lab/sensitive-data-detection.