We evaluate the psychometric reliability of Bayesian Knowledge Tracing for skill mastery classification. Using large-scale intelligent tutoring system data, we compare two simulation-based approaches for estimating classification consistency as reliability. Results support both approaches, show reliability increases with response-sequence length, and highlight the necessity of adjusting for chance agreement.