Affective Computing

We trained machine learning models to classify happy vs. sad moods in college students using data from surveys and wearable sensors. Factors such as poor health-related behavior, more academic activity hours, and more neutral social interactions were highly predictive of mood.

SNAPSHOT was a large-scale study of college undergraduates which tracked detailed longitudinal data from smartphones, wearable sensors, behavioral data, and mental health and sleep quality surveys. This initial study analyzed relationships between sleep quality, stress, and GPA, and used machine learning to predict these indices from objective phone and sensor data.

SmileTracker is an app that uses facial expression recognition to take a screenshot of the user’s screen whenever they smile. The screenshot and image of the user’s face are saved, to help them remember positive content they encountered during the day. Users can opt to share their images to a public gallery.

We propose a method for removing motion artifacts from Electrodermal Activity using a stationary wavelet transform. We modeled the wavelet coefficients as a Gaussian mixture distribution corresponding to the underlying skin conductance level and skin conductance responses. Our method achieves a greater reduction of artifacts while retaining motion-artifact-free data.

Using eye-tracking data collected while students interact with an Intelligent Tutoring System, we train machine learning models to predict when students are experiencing boredom and curiosity. Which analyze which features are most relevant to detecting when students are engaged and curious vs. disengaged and bored.

My Master’s Thesis investigated the usefulness of different data sources for automatically predicting when students using an Intelligent Tutoring System were engaged and curious, or disengaged and bored. Detailed comparisons of machine learning algorithms trained with eye-tracking data, Electrodermal Activity (EDA) and distance from the screen revealed that distance (which can be obtained with cheap infra-red sensors) provided one of the simplest and most reliable signals of student engagement.

An emerging field in user-adaptive systems is affect adaptivity: modeling and responding to an estimation of the user’s emotional state. Prior work used Dynamic Bayesian Networks to obtain adaptivity, but in this paper we represent the problem as a Partially Observable Markov Decision Process (POMDP) and find solutions that compute a plan of interventions for an Intelligent Tutoring System to take given an estimation of the user’s mood and goals.

This paper investigates the utility of using the Fast Johnson-Lindenstrauss Transform to produce a low-dimensional random projection of eye-tracking data features that can be used for classifying emotion in an Intelligent Tutoring System. Interestingly, the FJLT provides similar or superior performance to more computationally expensive techniques.

This study uses eye tracking to assess how students interact with automatic, adaptive hints in an Intelligent Tutoring System. Specifically, we study Prime Climb, an educational game which provides individualized support for learning number factorization skills in the form of hints generated from a model of student learning.