LLM autograders are usually evaluated against human “ground truth,” a single reference label provided by one annotator or constructed by aggregating multiple annotations. However, human grading is often variable and subjective. We instead propose to evaluate LLM graders by treating humans and LLMs as two grading systems and characterize their agreement and disagreement. We operationalize this approach using distance correlation and generalizability theory. Across two programming-grading datasets, we find higher Human–LLM agreement on more objectively specified rubric items. On items involving greater subjective judgment, LLMs exhibit strong internal consensus, diverging more from human graders.

Automated scene segmentation is an ongoing challenge in computational literary studies (CLS) to approach literary texts by analyzing comparable units. In this paper, we present our approach (work in progress) to text segmentation using a classifier that identifies the position of a scene change in English-language fiction. By manually annotating novels from a 20th-century US-English romance fiction corpus, we prepared training data for fine-tuning transformer models, yielding promising preliminary results for improving automated text segmentation in CLS.