We present an evaluation of several state-of-the-art machine translation systems supporting terminology constraints in the English–Finnish translation direction. We first perform a meta-evaluation, in which we critically evaluate the evaluation metrics we use, including the questions asked from human evaluators and the automatic evaluation methods. We find that common metrics such as term accuracy and TERm do not agree with the human evaluators’ judgement on the correctness of the terms, while LLM-as-a-judge shows promise even though it does not agree with the human evaluators on all questions. We then compare the evaluated systems based on the human evaluation results, LLM-as-a-judge, COMET, and chrF2. We find that of the systems considered, soft constraint methods, including a term-trained model and an LLM, perform better than hard constraints forced using a constrained beam search.

We present a hybrid rule-based and neural method for translating Finnish compound words into English. We use a lightweight set of rules to split a Finnish word into its constituent parts and determine the possible translations of those words using a dictionary. We then use an NMT model to rank these alternatives to determine the final output. Since the number of translations that takes into account different spellings, inflections, and word separators can be very large, we use beam search for the ranking when the number of translations is over a threshold. We find that our method is an improvement over using the same NMT model for end-to-end translation in both automatic and human evaluation. We conclude that our method retains the good qualities of rule-based translation such as explainability and controllability while keeping the rules lightweight.