3 - Decoding Language from the Brain

Summary

Abstract

In this paper we review recent computational approaches to the study of language with neuroimaging data. Recordings of brain activity have long played a central role in furthering our understanding of how human language works, with researchers usually choosing to focus tightly on one aspect of the language system. This choice is driven both by the complexity of that system, and by the noise and complexity in neuroimaging data itself. State-of-the-art computational methods can help in two respects: in teasing more information from recordings of brain activity and by allowing us to test broader and more articulated theories and detailed representations of language tasks. In this chapter, we first set the scene with a succinct review of neuroimaging techniques and what they have taught us about language processing in the brain. We then describe how recent work has used machine learning methods with brain data and computational models of language to investigate how words and phrases are processed. We finish by introducing emerging naturalistic paradigms that combine authentic language tasks (e.g., reading or listening to a story) with rich models of lexical, sentential, and suprasentential representations to enable an allround view of language processing.

Introduction

The study of language, like other cognitive sciences, requires of us to indulge in a kind of mind reading. We use a variety of methods in an attempt to access the hidden representations and processes that allow humans to converse. In formal linguistics intuitive judgments by the theorist are used as primary evidence – an approach that brings well-understood dangers of bias (Gibson and Fedorenko, 2010), but in practice can work well (Sprouse et al., 2013). Aggregating judgments over groups of informants is widely used in cognitive and computational linguistics, through both experts in controlled environments and crowdsourcing of naive annotators (Snow et al., 2008). Experimental psycholinguists have used a range of methods that do not rely on intuition, judgments, or subjective reflection, such as the speed of self-paced reading, or the order and timing of gaze events as recorded with eye-tracking technologies (Rayner, 1998).

Brain-recording technologies offer a different kind of evidence, as they are the closest we can get empirically to the object of interest: human cognition. Despite the technical challenges involved, especially the complexity of the recorded signals and the extraneous noise that they contain, brain imaging has a decades-long history in psycholinguistics.