Ecological Language: a multimodal approach to language learning and processing
The human brain has evolved the ability to support communication in complex and dynamic environments. In such environments, language is learned, and mostly used in face-to-face contexts in which processing and learning is based on multiple cues both linguistic and non-linguistic. Yet, our understanding of how language is learnt and processed comes for the most from reductionist approaches in which the multimodal signal is reduced to speech or text. I will introduce our current programme of research that investigates language in real-world settings in which learning and processing are intertwined and the listener/learner has access to -- and therefore can take advantage of -- the multiple cues provided by the speaker. I will then describe studies that aim at characterising the distribution of the multimodal cues in the language used by caregivers when interacting with their children (mostly 2-3 years old) and provide data concerning how these cues are differentially distributed depending upon whether the child knows the objects being talked about (allowing us to more clearly isolate learning episodes), and whether the objects are present (ostensive vs. non-ostensive). I will then move to a study using EEG addressing the question of how discourse but crucially also the non-linguistic cues modulate predictions about the next word in a sentence. I will conclude discussing the insights we have and (especially) can gain using this real world, more ecologically valid, approach to the study of language.