We present a test suite intended to study how well different machine translation systems do when translating syntactic constructions found in English that do not have a direct equivalent in Icelandic. We manually evaluate the translations and compare it to an automatic evaluation. Whereas the results demonstrate that the best-performing models achieve high accuracy in various categories, they still struggle with others.