Renjun Lai

Author directory

2026

Large language models are increasingly used as natural-language interfaces for structured information processing and agentic workflows, where user-specified constraints can determine downstream extraction, selection, tool use, and action planning. This paper studies whether models can execute natural-language concept-set constraints over candidate objects: given inclusion and exclusion conditions, the model must decide which candidates fall inside the intended concept boundary. We introduce a dual-resource diagnostic benchmark built from WordNet and BabelNet. The benchmark fixes backend constraints, candidate realizations, and gold answers before prompt rendering, and evaluates three task views: single-object judgment, exact candidate-set construction, and satisfiability judgment. The main evaluation set contains 202,500 rendered requests. Across 11 scored model runs, overall accuracy ranges from 43.70% to 56.36%, showing that the benchmark remains difficult for current strong models. Exact candidate-set construction yields consistently lower accuracy under stricter set-construction requirements, and recall diagnostics reveal asymmetric effects of constraint direction: increasing exclusion constraints lowers positive recall and raises negative recall, while increasing inclusion branches mainly erodes false-case rejection. These results suggest that concept-boundary execution is a measurable weakness under this diagnostic setup and that evaluating it requires factor-level diagnostics beyond aggregate accuracy.