The present paper focuses on the analysis of automatic subtitles produced with three different systems. We compare the outputs among each other paying attention to the categories of quality derived from audio-visual translation quality research. Besides that, we also consider comprehensibility of the produced subtitles. Additionally, we analyse the automatic evaluation scores to assess the overall quality. Our results show that automatically generated subtitles subtitles remain below human standards in quality and comprehensibility.