Reconstructing the Neural Code for Real World Face Perception

A central goal for neuroscience is to understand how our brains process information in real life, such as faces during natural social interactions. While we have gained important knowledge using simplified controlled experiments in laboratory settings, they cannot capture all aspects of how cognition works in the real world. Understanding the neurocognitive basis of natural behavior in humans requires both innovative data collection paradigms and statistical-computational models to understand real world brain-behavior relationships. We harnessed multi-electrode intracranial recordings from hours of unscripted interactions participants had with friends, family, experimenters, and others. We demonstrate that we can reconstruct both what participants saw on a fixation-by-fixation basis during natural social interactions based on the neural activity and accurately reconstruct the neural activity patterns based on the visual input using bidirectional, multivariate machine learning models. The results highlighted the social-vision pathway as particularly important to natural face perception. Sharper tuning was revealed for the type of facial expression over its intensity; and for subtler expressions and movements around a person’s resting expression versus strong expressions – a Weber’s law for facial expressions. These results suggest that oval-shaped neural tuning for the kind and intensity of dynamic facial expressions reflects the neural code for real-world face processing. This work illustrates a paradigm that allows us to both robustly model the neural underpinnings of real-world social perception and test and develop novel hypotheses about how we encode natural faces.

Learning Objectives:

1. Review the paradigms for studying real-world neuroscience.

2. Summarize the computational tools needed to model real-world variability to build robust statistical models.

3. Demonstrate knowledge about the neural code for real-world face perception.