Kristijan Armeni

Author directory

2026

Supervised sequence-labeling approaches to keyphrase extraction operate at token level, but whether a phrase is relevant may depend on context beyond its sentence. Processing an entire document at token resolution can increase sequence length and introduce irrelevant information. We investigate a mixed-resolution context-fusion architecture that retains token representations for the target sentence while compressing available surrounding sentences into pooled representations. Both representation types are placed in document order and processed jointly by a self-attention module before IOB prediction. We evaluate the model on \textsf{kp20k}, \textsf{inspec}, and \textsf{kptimes} against SEKE, a recent state-of-the-art supervised system with approximately 60% more total parameters. The model achieves comparable F1 scores on the two scientific-domain benchmarks, while a larger performance gap remains on the journalistic benchmark. Error analysis shows that the model generates substantially more candidates than there are gold keyphrases. However, many predictions counted as false positives under exact-match evaluation partially overlap with gold keyphrases, while exact matches tend to receive higher scores than partial and non-matches.

2024

To predict upcoming text, language models must in some cases retrieve in-context information verbatim. In this report, we investigated how the ability of language models to retrieve arbitrary in-context nouns developed during training (across time) and as language models trained on the same dataset increase in size (across scale). We then asked whether learning of in-context retrieval correlates with learning of more challenging zero-shot benchmarks. Furthermore, inspired by semantic effects in human short-term memory, we evaluated the retrieval with respect to a major semantic component of target nouns, namely whether they denote a concrete or abstract entity, as rated by humans. We show that verbatim in-context retrieval developed in a sudden transition early in the training process, after about 1% of the training tokens. This was observed across model sizes (from 14M and up to 12B parameters), and the transition occurred slightly later for the two smallest models. We further found that the development of verbatim in-context retrieval is positively correlated with the learning of zero-shot benchmarks. Around the transition point, all models showed the advantage of retrieving concrete nouns as opposed to abstract nouns. In all but two smallest models, the advantage dissipated away toward the end of training.

2022

When a language model is trained to predict natural language sequences, its prediction at each moment depends on a representation of prior context. What kind of information about the prior context can language models retrieve? We tested whether language models could retrieve the exact words that occurred previously in a text. In our paradigm, language models (transformers and an LSTM) processed English text in which a list of nouns occurred twice. We operationalized retrieval as the reduction in surprisal from the first to the second list. We found that the transformers retrieved both the identity and ordering of nouns from the first list. Further, the transformers’ retrieval was markedly enhanced when they were trained on a larger corpus and with greater model depth. Lastly, their ability to index prior tokens was dependent on learned attention patterns. In contrast, the LSTM exhibited less precise retrieval, which was limited to list-initial tokens and to short intervening texts. The LSTM’s retrieval was not sensitive to the order of nouns and it improved when the list was semantically coherent. We conclude that transformers implemented something akin to a working memory system that could flexibly retrieve individual token representations across arbitrary delays; conversely, the LSTM maintained a coarser and more rapidly-decaying semantic gist of prior tokens, weighted toward the earliest items.