Stance detection is widely used to summarize public discourse by measuring support, opposition, and neutrality towards societal issues. Evaluation pipelines rely on aggregate metrics such as macro-F1 that can mask how models actually distribute predictions across stance classes. We study this limitation by comparing large language and (statistical) supervised models across three stance detection datasets with different definitions of neutrality. Our evaluation considers predictive performance, class-level behavior, and environmental impact. We find that language models do not consistently outperform supervised baselines, and some models exhibit class-level priors that persist across datasets and prompting strategies. In addition, performance gains tend to saturate in the 4B–8B parameter range. These findings suggest that aggregate performance scores alone are insufficient for responsible stance detection. When model outputs are used to measure public opinion at scale, class-level behavior and deployment cost matter as much as the overall accuracy.