Communication and Language

Moral Foundations theory decomposes human moral reasoning into five factors, which vary reliably across different human populations and political affiliations. We use moral foundations to analyze large language models like GPT-3 to determine what, if any, consistent moral values it brings to conversations, whether these can be deliberately manipulated, and whether holding a particular moral stance affects downstream tasks.

We study the automatic generation of natural language navigation instructions in visually realistic indoor environments. Existing generators suffer from poor visual grounding, skip steps, and hallucinate objects. We address this using a large language model which incorporates visual landmark detection.. The model dramatically increases the quality of generated instructions, such that humans can follow them with a 71% success rate (SR); just shy of the 75% SR of real human instructions.

We train dialog models with interactive data from conversations with real humans, using a novel Offline RL technique based on KL-control. Rather than rely on manual ratings, we learn from implicit signals like sentiment, and show that this results in better performance.

Existing metrics for automatically evaluating dialog models correlate poorly with human judgements, and are evaluated on static conversation snippets. Instead, we deploy bots to interact live with humans, then approximate human ratings with state-of-the-art accuracy using conversations generated with self-play.

For the first time, we use hierarchical reinforcement learning to train open-domain dialog models, enabling the optimization of long-term, conversational, rewards, including reducing the toxicity of generated language. Our approach provides significant improvements over state-of-the-art dialog models.

My PhD Thesis spans both Social Reinforcement Learning and Affective Computing, investigating how affective and social intelligence can enhance machine learning algorithms, and how machine learning can enhance our ability to predict and understand human affective and social phenomena.

Social influence is a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning, through rewarding agents for having causal influence over other agents’ actions, thus increasing mutual information between agents’ actions. Optimizing for influence leads to agents learning emergent communication protocols. Unlike prior work, influence can be computed in a fully decentralized manner.

To combine supervised learning on data with reinforcement learning, we pre-train a supervised data prior, and penalize KL-divergence from this model using RL training. This enables effective learning of complex sequence-modeling problems for which we wish to match the data while optimizing external metrics like drug effectiveness. The approach produces compelling results in the disparate domains of music generation and drug discovery.

We collect observational data from real human conversations, and develop a measure of how much participants experienced bonding or chemistry. We analyze the effects of personality and attitudes on bonding, and find that attentiveness and excitement are more effective at promoting bonding than traits like attractiveness and humour.

Given only one-minute slices of facial expressions and body language, we use machine learning to accurately predict whether two humans having a conversation will bond with each other. We analyze factors which lead to bonding and discover that synchrony in body language and appropriate, empathetic facial expressions lead to higher bonding.