Semantic Composition and the Brain

I write this from the European Society for Philosophy and Psychology meeting taking place at the University of Hertfordshire this year (August 14-August 17). The keynote on the first day was delivered by Prof. Liina Pylkkänen (NYU) talking about semantic composition in the brain. Liina is known to many theoretical linguists for her hugely influential dissertation on verbal argument structures (Introducing Arguments, MIT 2002), but Liina has been following up that theoretical interest in the syntax/semantics interface with a research programme looking at the neural underpinnings of language. Her hugely ambitious goals have been facilitated by the fact that our techniques for measuring brain activity have become much better and more fine grained in the last 25 years. However, this kind of research is still devilishly difficult to do and get replicable and robust effects, and even more difficult to interpret. Her research has focused on the technique of magnetoencephalography (MEG), directing the methodology on the problem of semantic composition specifically. MEG is a nice choice of technique because it provides fine grained spatial resolution for brain effects (unlike EEG) as well as finegrained information about the time course of those effects (unlike fMRI).

In theoretical linguistics, it is often assumed that neurolinguistics is still too remote from theoretical concerns to be directly relevant to it (see Embick and Poeppel 2015 on the commensurability gap), and so the communication between the two subfields is intermittent and sporadic. And to be honest, it is so hard to keep up just with new developments in one’s own small corner! But this is where general and crossdisciplinary conferences come in, and why they are so important.

Specifically, it is great to hear a neurolinguistic keynote with a state of the art update on what has been discovered. In the case of Pylkkänen’s work the experience is richly rewarding, and contains much food for thought. I summarize some of the main points that emerged from her talk here, in case it motivates some people to dip into her research.

Pylkkänen takes a deliberately bottom up approach, targeting core robust cases of semantic combination and devising families of minimal pairs to test the relevance of different factors in the activation of certain areas. The big picture that emerges is the existence of a pair of activation areas, one early in the lateral anterior temporal lobe (LATL) (200 ms) and one a bit later in the ventro-medial prefrontal cortex (vmPFC).  The LATL is particularly interesting because it is very early in comprehension, and it is differentially implicated in both  production (see here) and comprehension (see here), both reading and auditory of complex phrases such as red car as compared to control conditions consistingjust of two unconnected words in a list (Bemis & Pylkkänen, 2011, 2013a,2013b)).

The LATL has been implicated differentially in the processing of sentences as opposed tolists of words in a huge amount of literature (Friederici et al., 2000; Humphries, Binder, Medler, & Liebenthal, 2006; Humphries, Love, Swinney, & Hickok, 2005; Jobard, Vigneau, Mazoyer, & Tzourio- Mazoyer, 2007; Mazoyer et al., 1993; Pallier, Devauchelle, & Dehaene, 2011; Rogalsky & Hickok, 2009; Stowe et al., 1998; Vandenberghe, Nobre, & Price, 2002), but since sentence processing involves a host of different processes, we still don’t know what specific job the LATL is doing, which is why Pylkkänen’s lab has been focusing on simple two word combinations in their own series of experiments.

In one set of experiments (Blanco-Elorrieta, E., Kastner, I.,Emmorey, K. & Pylkkänen, L.`A shared neurobiology for building phrases in signed and spoken language’ (submitted)), the way that the two conditions were controlled for while keeping the stimulus as similar as possible was quite ingenious. (Because brain activity is complex and affected all the time by other things not being tested by the experimenter, it is very important in these methodologies to keep the stimulus as similar as possible, while manipulating the test factors.) In a production task, subjects were asked to name pictures of simple objects coloured in in simple block colours. The coloured objects in the pictures were also however situated against backgrounds of different colours. In one run type, the subject was asked to name the object with its colour (e.g. red car) and in the other type of run, the instructions were to give the name of the background colour followed by giving the name of the object. While the second condition is plausibly harder and less natural, the complexity goes in the other direction than the effect sought. Indeed, there was found to be significantly more activation of LATL in the phrasal composition case as compared to the list. The effect was found also across languages (for Arabic see here) (and across modes of articulation (for ASL). The effect was found in both the nominal and the verbal domain (see Westerlund, M., Kastner, I., Al Kaabi, M., & Pylkkänen, L. (2015) ).

It however is even more specific than this. It is specifically involved in the combination of concepts and not syntactic or semantic combination more generally (Del Prato & Pylkkänen, 2014; Westerlund & Pylkkänen, 2014; Zhang & Pylkkänen, 2015). For example, it is found in the early computation of red car but not in the computation of    two cars (Del Prato  \& Pylkkänen, 2014).  It also appears to be an interaction between the specificity of the left member of the phrase when it needs to be integrated with a noun (more specific left members as in tomato soup producing greater activity than compounds with vaguer left members (vegetable soup) (Zhang and Pylkkänen 2015 )

There is also a contrast between the following two cases:

(1) a. The girls are tall and blonde.

b. The girls are Finnish and Dutch.

In a context where all four girls are tall and blonde, but only two are Finnish and two are Dutch, both (1-a) and (1-b) are of course true on an intersective vs. collective reading of the adjectival conjunction. It turns out that the LATL effect only emerges in (1-a) and not (1-b) (Poortman and Pylkkänen 2016 ).

So what is this LATL area and what is it doing? Why is it activated so early, immediately at the onset of lexical access? It does not generally kick in whenever there is syntactic combination, but is tied specifically to the kind of concept formation that we would normallly think of as being low in the hierarchical linguistic representation. How do we characterize in more theoretical terms the kind of semantic composition that is going on very early in this area of the brain. This is an example of a robust and replicated effect in a specific brain area, which I would argue should start to inform our theories on the architecture of the language faculty.

If indeed we believe theoretical linguistics is ready to have that conversation which our sister disciplines. Fields that call themselves ‘biolinguistics’ certainly seem to think that theoretical linguistics is ‘ready’. But being ready does not just mean being willing to speculate about evolutionary origins or invoke ‘third factor’ principles. I mean : are we ready to be able to do something with facts like these from the literature? Maybe we’re not ready. But if so, then let’s not call what we do biolinguistics just yet.

Actually I do hope we are ready. I think our field is progressing fast in this domain and we are at the beginning of a very exciting time.