Large language models (LLMs) have demonstrated exceptional performance across various domains. However, their increasing sizes pose significant deployment challenges, particularly in resource-constrained scenarios. While network pruning provides an effective compression strategy, existing approaches either depend on calibration datasets—leading to poor generalization—or require training auxiliary predictors, incurring significant computational overhead. In this paper, we propose DSPP (Dataset-Specific Progressive Pruning), a novel training-free structured pruning method that dynamically updates the importance scores of neurons during inference to identify and retain dataset-specific neurons. By employing a progressive pruning strategy, DSPP can further reduce computational overhead. Experimental results show that DSPP demonstrates remarkable generalization capabilities across diverse datasets and models, while achieving a 1.36× speedup at 50% sparsity level.