Accounting for disagreement among annotators is important for a variety of NLP applications. However, understanding what insights can be drawn from disaggregated annotations remains an open question. We introduce FLOCKE, a framework that jointly learns a task and models latent groups of annotators with similar behavior. We first formalize six analyses enabled by our framework, including annotator-group discovery, group- and individual-level label prediction, and finding the items most prone to cause disagreement. We then validate FLOCKE with a controlled synthetic benchmark, creating predefined annotator groups and labeling patterns, and showing that it can recover group structure with very high accuracy. We apply FLOCKE to two real-world datasets, where it predicts individual annotations, discovers interpretable annotator groups, and shows that groups are not explained by socio-demographic traits alone.