Generative commonsense reasoning (GCR) requires models to synthesize coherent narratives that simultaneously satisfy lexical constraints and commonsense logic. Although ensemble-based LLM strategies are widely adopted to alleviate the fragility of single-chain reasoning, we uncover a counterintuitive homogeneity trap in GCR. Specifically, we observe that increasing the number of reasoning chains can degrade performance, as the generated chains tend to collapse into a narrow semantic region, thereby reinforcing shared biases rather than providing complementary evidence. We posit that escaping this trap requires fundamentally broadening semantic coverage via heterogeneous sources. Our investigation into the nature of diversity reveals that deep semantic diversity, rather than surface-level lexical variation, is the decisive prerequisite for effective integration. Motivated by this insight, we propose an Explore-then-Integrate framework, in which high–semantic-entropy explorers capture diverse concept bindings, and a powerful integrator performs compositional synthesis to merge valid fragments into coherent narratives. Crucially, to ensure that the observed performance gains arise from accurate logical composition rather than trivial best-candidate selection, we introduce a provenance-aware evaluation suite that explicitly quantifies the heterogeneous origins of synthesized outputs. Extensive experiments on multiple benchmarks demonstrate the consistent superiority of our approach across a range of metrics. Notably, our method achieves over 10% improvement in overall accuracy on NoRa and in SPICE score on CommonGen-Lite.

Hypernetworks are a class of meta-networks that generate weights for main neural networks. Their unique parameter spaces necessitate exploring suitable optimization strategies to enhance performance, especially for language models. However, a comprehensive investigation into optimization strategies for hypernetworks remains absent. To address this gap, we analyze the loss landscape of hypernetworks and propose that restart optimization strategies can improve their performance for language models. We find that hypernetworks have inherently more complicated loss landscapes compared to conventional networks due to their distinct parameter spaces. Consequently, a restart strategy that periodically resets the learning rate can facilitate better convergence for hypernetworks. Through experiments on instruction tuning and multi-task training, we demonstrate that the restart strategy consistently enhances the performance of hypernetworks for language models, often more effectively than for conventional deep neural networks. Our findings highlight the importance of tailored optimization techniques to unlock the full potential of hypernetworks in natural language processing tasks.