Human annotators typically assess translation quality by considering the broader document context, which allows them to detect errors arising from discourse-level phenomena that may be missed when evaluating text segments in isolation. However, the role of document-level context in automatic Machine Translation (MT) evaluation remains underexplored. In this work, we present a controlled study of the impact of document context on LLM-based MT evaluation and its interaction with the availability of manually curated references. We consider several evaluation configurations, also exploring how performance varies when enabling the thinking inference mode and conducting task-specific fine-tuning. Our results show that the effect of document context differs substantially across test sets, and that input repetition itself can account for part of the apparent gains. Fine-tuning improves performance consistently and offers larger gains when document context is available, suggesting that task-specific adaptation may help models to handle document-level context more effectively.