The emergence of open data portals necessitates more attention to protecting sensitive data before datasets get published and used. To do so effectively, we observe the need to refine and broaden our definitions of sensitive data. We argue that the sensitivity of data depends on its context, and that this context should be considered in methods for assessing data sensitivity. To this end, we introduce a framework, termed ConSenser, for contextual sensitive data detection based on two aspects: 1) type contextualization, which considers the type of data values within the overall context of the dataset or document to assess their true sensitivity, and 2) domain contextualization, which assesses the sensitivity of data values informed by domain-specific information external to the dataset, such as geographic origin of a dataset. Experiments instrumented with language models confirm that type-contextualization significantly reduces the number of false positives for type-based sensitive data detection and reaches a recall of 94% compared to 63% with commonly used tools. Moreover, domain-contextualization through retrieval of sensitivity rules effectively grounds sensitive data detection in relevant context in non-standard data domains. A case study with humanitarian data experts also illustrates that context-grounded explanations provide useful guidance in manual data auditing processes. We open-source the code and annotated datasets at https://github.com/trl-lab/sensitive-data-detection.

Question answering in natural language over tabular and relational data has become a key capability in the democratization of insights from structured data sources. These systems first require table retrieval to identify data that is relevant to a given natural language query, for which several methods have been introduced. In this work we present and study a fine-grained decomposition-based and global connectivity-aware table retrieval mechanism (DCTR) that handles the challenges induced by open-domain question answering over relational databases in complex usage contexts. We evaluate the effectiveness of these two mechanisms through the lens of retrieval complexity which we measure along the axes of query- and data complexity. Our complexity analyses over industry-aligned benchmarks illustrates the robustness of DCTR for highly composite queries and densely connected relational databases.

Large Language Models (LLMs) excel in natural language tasks, but less is known about their reasoning capabilities over tabular data. Prior analyses devise evaluation strategies that poorly reflect an LLM’s realistic performance on tabular queries. Moreover, we have a limited understanding of the robustness of LLMs towards realistic variations in tabular inputs. Therefore, we ask: Can general-purpose LLMs reason over tabular data, really?, and focus on two questions 1) are tabular reasoning capabilities of general-purpose LLMs robust to real-world characteristics of tabular inputs, and 2) how can we realistically evaluate an LLM’s performance on analytical tabular queries?Building on a recent tabular reasoning benchmark, we first surface shortcomings of its multiple-choice prompt evaluation strategy, as well as commonly used free-form text metrics such as SacreBleu and BERT-score. We show that an LLM-as-a-judge procedure yields more reliable performance insights and unveil a significant deficit in tabular reasoning performance of LLMs. We then extend the tabular inputs reflecting three common characteristics in practice: 1) missing values, 2) duplicate entities, and 3) structural variations. Experiments show that the tabular reasoning capabilities of general-purpose LLMs suffer from these variations, stressing the importance of improving their robustness for realistic tabular inputs.