Embedding representations are widely used in natural language processing, yet their semantic interpretation depends on the operators used to compare them. Previous work has shown that, for certain embedding models, the inner product is linearly correlated with Pointwise Mutual Information (PMI). However, this relation has so far been established only empirically or analytically for specific models.

In this work, we formally show that, under certain assumptions about distributional semantic embedding models, the inner product between embeddings approaches PMI. This theoretical result provides a more general explanation for the observed relationship and enables the formulation of training criteria aimed at learning embedding spaces in which operations between embeddings are semantically interpretable.