M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction

Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal

Collaborative Robotics Lab, University of Virginia

Code Paper

Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot systems often treat robots as functionally interchangeable, overlooking how distinct agent identities shape user perception and how such individuality changes the coordination requirements of multi-robot interaction. To address this, we introduce M2HRI, a multimodal multi-agent framework that models each robot as an identity-bearing agent through personality and long-term memory, together with a contextualized coordination mechanism that regulates agent participation. In a controlled user study (n = 105) in a multi-agent human–robot interaction (HRI) scenario, we found that most personality contrasts were distinguishable and consistently expressed. Long-term memory improved preference awareness and interaction naturalness, while contextualized coordination improved conversational flow, response appropriateness, and overlap avoidance. Together, these findings show that agent individuality and contextualized participation coordination play complementary roles in supporting coherent and socially appropriate multi-agent HRI.

Multimodal Multi-Agent Interaction

A human user interacts with two NAO robots, each with distinct personality, memory, and perception, enabling personalized and context-aware embodied interaction.

Our Approach: M2HRI

M2HRI is a multimodal multi-agent HRI framework that supports interaction with a team of identity-bearing robot agents. Each robot maintains its own personality and long-term memory, while a contextualized coordinator regulates which agent should participate based on the interaction context and each agent’s characteristics. This design preserves agent individuality while producing coherent group behavior.

Personality Big Five traits are integrated into each agent’s reasoning process, shaping how the robot responds, expresses itself, and maintains a consistent identity.
Memory Each agent maintains working memory for short-term context and semantic and episodic memory for user-specific knowledge and prior interaction history.
Perception Speech and visual observations are jointly processed to ground user input in the physical scene and support context-aware reasoning.
Planning An LLM-based planner generates sequenced speech, gesture, and movement policies conditioned on perception, personality, memory, and available robot actions.
M2HRI diagram
Multi-agent coordination diagram
Coordination A contextualized coordinator scores each agent’s participation suitability using shared context, local observations, and agent identity, selecting which robot should respond.

Findings & Insights

We evaluated M2HRI through a controlled user study with 105 participants across seven experimental blocks, varying personality, memory, and coordination.

1

Personality supports agent individuality

Participants generally distinguished between robot personalities, recognized intended traits, and perceived personality expression as consistent across interactions. The effect varied by trait, with some traits more behaviorally salient than others.

Figure (right): Personality evaluation results (RQ1). Mean Likert ratings (±1 SD) for (a) distinguishability, (b) consistency, and (c) engagement across five Big Five trait conditions (O = Openness, C = Conscientiousness, E = Extraversion, A = Agreeableness, N = Neuroticism). The dashed line indicates the neutral midpoint (μ0 = 3.0); stars denote significance against μ0 via one-sample t-tests. (d) Personality Trait recognition accuracy; the dashed line indicates chance level (33.3%).

Personality findings
2

Memory enables personalization

Long-term memory improved recall accuracy, preference awareness, and perceived personalization. Without long-term memory, agents remained locally coherent but impersonal.

Figure (right): Paired bar charts compare with- and without-condition means (±1 SD) across three measures each. Memory measures: (a) recall accuracy, (b) preference awareness, (c) naturalness. Brackets indicate pairwise significance via paired-sample t-tests with ** p < .01, *** p < .001.

Memory findings
3

Coordination structures multi-agent interaction

Contextualized coordination improved conversational flow, response appropriateness, and overlap avoidance. These results show that multi-agent turn-taking requires deciding which agent should respond, not only when a response should occur.

Figure (right): Paired bar charts compare with- and without-condition means (±1 SD) across three measures each. Coordination measures: (d) conversational flow, (e) response appropriateness, (f) overlap avoidance. Brackets indicate pairwise significance via paired-sample t-tests with ** p < .01, *** p < .001.

Coordination findings

Interaction Examples

Personality: Two robots with contrasting personality configurations respond differently to the same situation, making agent identity perceptible through language, tone, and behavior.

Memory: Robots recall and reuse user-specific preferences from prior interactions, supporting more personalized and context-aware responses.

Coordination: Robots regulate participation so the contextually appropriate agent responds, reducing interruptions and maintaining smoother multi-agent dialogue.