We used confirmatory factor analysis to assess the reliability and construct representation of an LLM-based measurement instrument of language proficiency. LLMs were at least as reliable as human raters and loaded onto the same underlying factor, though analyses indicated a less than perfect alignment between LLM and human raters.