Text-to-chart retrieval enables users to locate relevant charts through natural language queries and has emerged as a critical capability in modern business intelligence systems. However, current benchmarks fail to simulate realistic user queries or test deep semantic understanding, leaving a significant evaluation gap. To address this, we introduce CRBench, the first real-world BI-sourced benchmark comprising 21,862 charts and 326 queries, which adopts a Target-and-Distractor paradigm to evaluate discriminative retrieval among highly similar candidates. Benchmarking reveals that existing methods, which rely primarily on visual features, fail to capture the rich analytical semantics of charts. We therefore propose a semantic insights synthesis pipeline that automatically generates three hierarchical levels of insights for alignment training: visual patterns, statistical properties, and practical applications. Using this pipeline, we produce 207,498 insights for 69,166 charts as training data. Leveraging this data, we develop ChartFinder, which significantly outperforms state-of-the-art methods on CRBench, achieving 66.9% NDCG@10 for precise queries (an 11.58% improvement) and an average gain of 5% across fuzzy query metrics.