Understanding human beliefs and social attitudes often relies on carefully designed survey instruments. Recent work has suggested that large language models (LLMs) could automate parts of this process by generating surveys at scale, raising questions about the comparability of such instruments to literature-grounded, human-designed surveys. We present a controlled empirical comparison between GPT-generated surveys and established survey baselines across three social domains: climate change, immigration, and diversity, equity, and inclusion (DEI). GPT-generated surveys were produced using a fixed prompting framework enforcing a 3×3 structure over beliefs, perceptions, and behaviors, while human baselines were assembled from validated instruments to match survey length and construct coverage. We collected responses from U.S.-based participants, who completed both survey types, allowing direct within-subject comparison. We analyze differences in response distributions, clustering behavior, and alignment with self-identified stances. Our results show that GPT-generated surveys capture the same dominant attitudinal divisions as human-designed instruments, while exhibiting differences in the resolution of belief structure and group separation. These findings suggest that LLM-generated surveys are suited for exploratory and large-scale analyses, and can be used to complement expert-designed instruments.

Large Language Models (LLMs) are sensitive to the order in which answer choices are presented in Multiple-Choice Question Answering. The same question can yield a different prediction when its options are shuffled. Although exhaustive majority voting over all permutations can mitigate this bias, its factorial cost makes it impractical at inference time. We propose GPS (Guided Permutation Scoring), a two-stage inference-time pipeline that selects option orderings more likely to produce correct predictions from a frozen LLM. A sensitivity classifier first identifies which questions are genuinely vulnerable to positional interference; robust questions are answered directly, while sensitive ones are forwarded to a trained permutation scorer that identifies a more position-stable option ordering, promoting better predictions. Unlike prior approaches that either enumerate all permutations uniformly, or require modifying the LLM’s parameters, GPS is the first method to actively select permutations so that the base model can consistently give better answers. GPS steadily improves Accuracy, and Macro F1 across all evaluated model-dataset combinations, with remarkable reductions in positional bias metrics.