Black-box knowledge tracing models are commonly deployed to drive personalization in online learning platforms and are typically evaluated using classification metrics such as AUC, accuracy, and F1 score. However, a model that predicts item responses using only each item’s proportion correct in the training set achieves AUC up to 0.72 and accuracy up to 0.84 on widely used benchmark datasets, despite using no information about individual students’ response histories. Inspired by the concept of marginal reliability in psychometrics, we introduce Fractional Information Gain (FIG), an information-theoretic evaluation metric for black-box predictive models of student item responses. FIG measures the fraction of a student’s response uncertainty resolved by a trained model relative to the item-only baseline. FIG equals 0 when the model adds no information beyond item base rates, and 1 when held-out responses are perfectly predicted. FIG is applicable to any model that outputs probabilities and is sensitive to calibration errors. We characterize FIG on synthetic and real data, compare it to AUC for both the item-only baseline and trained models on four benchmark datasets, and describe the operational affordances that FIG inherits from its reliability-like construction. FIG imports the conceptual benefits of score reliability into the prediction-oriented framework of ML-driven educational systems.