Backpack language model embeddings represent each word as a weighted sum of sense vectors, offering more interpretable and intervenable lexical representations than standard contextual embeddings. Backpack sense vector representations show strong performance on lexical similarity tasks in a monolingual setting, as well as cross-lingual transfer through finetuning on aligned parallel multilingual data. Our work investigates whether representations from Backpack models trained from scratch on parallel aligned multilingual data allow for interpretable control without sacrificing translational and cross-lingual representation learning ability. We train matched 131M parameter Transformer and 132M parameter Backpack models from scratch on parallel aligned English-French EuroParl and evaluate their expressivity using validation perplexity, their learned monolingual and cross-lingual lexical relationships using Multi-SimLex and their translational ability using sentence-level cosine similarity. While the Transformer attains higher English and cross-lingual Multi-SimLex, the Backpack outperforms the Transformer on French Multi-SimLex and shows stronger translated-vs-random sentence separation. We finally compare sense-level gender bias ablations on the Backpack model with nullspace projection debiasing on the Transformer, demonstrating the effectiveness of sense vector control for debiasing multilingual representations.