In this paper, we present a systematic survey and methodological guidelines for explanation evaluation and statistical significance testing in explainable NLP (XAI). We systematically reviewed 494 verified papers from the ACL Anthology (2020–2025) to identify how explanations are evaluated and the frequency of statistical validation. Our analysis identifies a persistent **rigour gap**: although **the number of explainable NLP papers increased fivefold** during the study period, there was **no corresponding increase in the use of explanation-specific statistical testing**. Across the full corpus, only 25.30% (125/494) of studies substantiate evaluation claims with hypothesis testing. This remains a minority practice in every surveyed year. We categorise the primary barriers to rigour into three issues: implicit generalisation, inferential reporting gap, and incomplete rigour. To address these, we propose a bipartite framework that maps research hypotheses to appropriate statistical testing regimes. Our goal is to clarify when statistical testing is necessary and to provide a guideline for more rigorous, reproducible explainability research in the LLM era.