Mixture of Experts (MoE) has great potential in scaling up the capacity of large models while maintaining low computational costs. However, existing methods mainly reduce redundancy at the expert level while overlooking fine-grained neuron-level redundancy and expert collaboration. In this paper, we propose an Information Bottleneck based MoE (IBMoE) for parameter-efficient fine-tuning, which reduces neuron-level redundancy within each expert and fosters internal collaboration among all experts. Specifically, IBMoE equips each expert with learnable queries to capture task-specific characteristics and employs sparse neuronal activation to dynamically select the most relevant neurons, reducing redundant activations across tasks. In addition, a diversity constraint is imposed among experts to maximize their knowledge diversity, enabling them to cooperate more effectively. Extensive experiments across multiple benchmarks show that IBMoE consistently outperforms strong baselines while activating significantly fewer parameters. In addition, IBMoE reduces inference latency by 61.7% and the memory footprint of adapters by 48.2% compared with previous MoE-based adaptation methods, while achieving comparable training costs.