Personality is a key factor in human–agent and human-robot interaction, influencing user perception, trust, and engagement. This work investigates methods for inducing Big Five personality traits in (open) language models, with a focus on smaller variants as more easily deployable in real-time agents and robotic systems. We compare prompt-based conditioning, parameter-efficient fine-tuning, and activation steering, focusing on their behaviour in multi-turn dialogue and embodied gesture generation. In particular, we investigate personality drift across turns and non-verbal behaviour generation through direct function calls.

We find that classifier-assigned trait alignment differs little across methods, with no consistent evidence of trait drift over multi-turn interaction. However, conditioning method significantly affects dialogue quality and robustness, with steering and fine-tuning introducing artefacts and instability. Interestingly, the more consequential differences emerge in the multimodal channel. Conditioning method, not intended personality, is the dominant driver of gesture behaviour, reshaping gesture diversity, tool-use reliability, and output-gesture congruence. Our findings suggest that personality conditioning should be evaluated jointly across verbal and non-verbal channels, and that method choice is itself a design decision shaping the expressiveness and reliability of embodied conversational agents.

Adapting large language models to user-specific preferences is often constrained by the cost of human annotation, making preference optimisation impractical in low-resource settings where preferences cannot be reliably labelled by LLMs themselves, e.g., due to cultural, subjective, or personalised contexts. In this paper, we investigate how language models encode preference information in their intermediate representations, finding that activations from chosen and rejected responses form distinct clusters across layers, even in pretrained models. Strikingly, this structure is strengthened by alignment on canonical datasets but erased when the target preferences differ from those the model was aligned on, suggesting aligned LLMs are poor judges for non-mainstream populations. Exploiting this structure, we propose training a lightweight linear probe on a few labelled preference pairs (≤500) and using it to annotate large unlabelled datasets (50K+) for downstream preference optimisation. We systematically evaluate this approach across different datasets, preference optimisation methods and model scales and find that our method consistently outperforms direct training given the same annotation budget, and remains competitive against baselines trained on 50-100× more labelled data in the majority of our settings.

Fine-tuning Large Language Models (LLMs) with first-order methods like back-propagation is computationally intensive. Zeroth-Order (ZO) optimisation uses function evaluations instead of gradients, reducing memory usage, but suffers from slow convergence in high-dimensional models. As a result, ZO research in LLMs has mostly focused on classification, overlooking more complex generative tasks. In this paper, we introduce ZOPrO, a novel ZO algorithm designed for Preference Optimisation in LLMs. We begin by analysing the interplay between policy and reward models during traditional (first-order) Preference Optimisation, uncovering patterns in their relative updates. Guided by these insights, we adapt Simultaneous Perturbation Stochastic Approximation (SPSA) with a targeted sampling strategy to accelerate convergence. Through experiments on summarisation, machine translation, and conversational assistants, we demonstrate that our method consistently enhances reward signals while achieving convergence times comparable to first-order methods. While it falls short of some state-of-the-art methods, our work is the first to apply Zeroth-Order methods to Preference Optimisation in LLMs, going beyond classification tasks and paving the way for a largely unexplored research direction. Code and visualisations are available at https://github.com/alessioGalatolo/VisZOPrO.