Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. This leaves a complementary but under-examined setting: scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit object naming alone. We introduce Referring Scenario Comprehension (RSC), a benchmark designed to study this setting. RSC queries describe roles, goals, and contextual cues, and deliberately reference distractor objects to reduce lexical shortcutting. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position, which expose distinct failure modes and support fine-grained analysis. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories to test generalization beyond category co-occurrence. We further propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining supervised warm-starting with difficulty-aware reinforcement learning. Experiments show that scenario-based queries expose systematic shortcut failures in current models that standard benchmarks do not reveal, and that curriculum training improves performance on challenging slices and transfers robust improvement to standard benchmarks.