1

Hierarchical Reinforcement Learning for Open-Domain Dialog

For the first time, we use hierarchical reinforcement learning to train open-domain dialog models, enabling the optimization of long-term, conversational, rewards, including reducing the toxicity of generated language. Our approach provides significant improvements over state-of-the-art dialog models.

A. Saleh*, Natasha Jaques*, A. Ghandeharioun, J. H. Shen, R. Picard

2019 In Association for the Advancement of Artificial Intelligence (AAAI) Oral (top 7.8% of submissions)

Hierarchical Reinforcement Learning for Open-Domain Dialog

Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

Social influence is a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning, through rewarding agents for having causal influence over other agents’ actions, thus increasing mutual information between agents’ actions. Optimizing for influence leads to agents learning emergent communication protocols. Unlike prior work, influence can be computed in a fully decentralized manner.

Natasha Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. A. Ortega, D. J. Strouse, J.Z. Leibo, N. de Freitas

2019 In International Conference on Machine Learning (ICML) Best Paper Honourable Mention (top 0.26% of submissions)

Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

Tackling Climate Change with Machine Learning

This paper comprehensively surveys the ways in which machine learning could be usefully deployed in the fight against climate change. From smart grids to disaster management, we identify high impact problems and outline how machine learning can be employed to address them.

D. Rolnick, P. L. Donti, L. H. Kaack, K. Kochanski, A. Lacoste, K. Sankaran, A. S. Ross, N. Milojevic-Dupont, Natasha Jaques, A. Waldman-Brown, A. Luccioni, T. Maharaj, E. D. Sherwin, S. K. Mukkavilli, K. P. Kording, C. Gomes, A. Y. Ng, D. Hassabis, J. C. Platt, F. Creutzig, J. Chayes, Y. Bengio

2019 In ACM Computing Surveys

Tackling Climate Change with Machine Learning

Learning via Social Awareness: Improving a Deep Generative Sketching Model with Facial Feedback

We show the outputs of a generative model of sketches to human observers and record their facial expressions. Using only a small number of facial expression samples, we are able to tune the model to produce drawings that are significantly better rated by humans.

Natasha Jaques, J. McCleary, J. Engel, D. Ha, F. Bertsch, D. Eck, R. Picard

2018 In International Conference on Learning Representations (ICLR) workshop

Learning via Social Awareness: Improving a Deep Generative Sketching Model with Facial Feedback

Vomit Comet Physiology: Autonomic Changes in Novice Flyers

During a zero-gravity parabolic flight, we recorded participants’ heart rate, accelerometer, and skin conductance measurements as well as their self-report nausea, anxiety, and excitement. Statistical analysis revealed that skin conductance is predictive of nausea, while heart rate is predictive of anxiety and excitement.

K. Johnson, S. Taylor, S. Fedor, Natasha Jaques, W. Chen, R. Picard

2018 In IEEE Engineering in Medicine and Biology Society (EMBC)

Vomit Comet Physiology: Autonomic Changes in Novice Flyers

Multimodal Autoencoder: A Deep Learning Approach to Filling in Missing Sensor Data and Enabling Better Mood Prediction

Predicting signals like stress and health depends on collecting noisy data from a number of modalities, e.g. smartphone data, or physiological data from a wrist-worn sensor. Our method can continue making accurate predictions even when a modality goes missing; for example, if the person forgets to wear their sensor.

Natasha Jaques, S. Taylor, A. Sano, R. Picard

2017 In International Conference on Affective Computing and Intelligent Interaction (ACII)

Multimodal Autoencoder: A Deep Learning Approach to Filling in Missing Sensor Data and Enabling Better Mood Prediction

Personalized Multitask Learning for Predicting Tomorrow's Mood, Stress, and Health

Traditional, one-size-fits-all machine learning models fail to account for individual differences in predicting wellbeing outcomes like stress, mood, and health. Instead, we personalize models to the individual using multi-task learning (MTL), employing hierarchical Bayes, kernel-based and deep neural network MTL models to improve prediction accuracy by 13-23%.

Natasha Jaques*, S. Taylor*, E. Nosakhare, A. Sano, R. Picard

2017 In IEEE Transactions on Affective Computing (TAFFC) Best Paper; NeurIPS Machine Learning for Healthcare (ML4HC) Workshop Best Paper

Personalized Multitask Learning for Predicting Tomorrow's Mood, Stress, and Health

Predicting Tomorrow’s Mood, Health, and Stress Level using Personalized Multitask Learning and Domain Adaptation

Modeling measures like mood, stress, and health using a monolithic machine learning model leads to low prediction accuracy. Instead, we develop personalized regression models using multi-task learning and Gaussian Processes, leading to dramatic improvements in next-day predictions.

Natasha Jaques, O. Rudovic, S. Taylor, A. Sano, R. Picard

2017 In Proceedings of Machine Learning Research

Predicting Tomorrow’s Mood, Health, and Stress Level using Personalized Multitask Learning and Domain Adaptation

Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control

To combine supervised learning on data with reinforcement learning, we pre-train a supervised data prior, and penalize KL-divergence from this model using RL training. This enables effective learning of complex sequence-modeling problems for which we wish to match the data while optimizing external metrics like drug effectiveness. The approach produces compelling results in the disparate domains of music generation and drug discovery.

Natasha Jaques, S. Gu, D. Bahdanau, J. M. Hernandez-Lobato, R. E. Turner, D. Eck

2017 In International Conference on Machine Learning (ICML)

Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control

BITxBIT: Encouraging Behavior Change with N=2 Experiments

To help promote behavior change, we leverage the power of social obligation, and conduct an experiment in which participants are paired together and asked to design a Behavioral Intervention Technology (BIT) customized to suit their partner’s behavior change goal.

Natasha Jaques, T. Rich, K. Dinakar, N. Farve, W.V. Chen, P. Maes, R. Picard

2016 In Proceedings of the CHI Conference Extended Abstracts on Human Factors

BITxBIT: Encouraging Behavior Change with N=2 Experiments