4 - Integrated Model of Text and Picture Comprehension

Summary

Abstract

This chapter presents an integrated model of text and picture comprehension that takes into account that learners can use multiple sensory modalities combined with different forms of representation. The model encompasses listening comprehension, reading comprehension, visual picture comprehension, and auditory picture comprehension (i.e., sound comprehension). The model’s cognitive architecture consists of modality-specific sensory registers, working memory, and long-term memory. Within this architecture, a distinction is made between perception-bound processing of text surface or picture surface structures, on the one hand, and cognitive processing of semantic deep structures, on the other hand. The perception-bound processing of text surface structures includes phonological and graphemic input analyses with graphemic–phonemic conversion, leading to lexical patterns. The perception-based processing of picture surface structures includes visual or acoustic nonverbal feature analyses, leading to visuospatial patterns or sound patterns. The cognitive processing includes descriptive processing of lexical patterns (via parsing), which leads to propositional representations, and depictive processing of spatial or sound patterns (via structure mapping), which leads to mental models. Propositional representations and mental models interact via model construction and model inspection processes. After presenting the integrated model of text and picture comprehension, the chapter derives predictions, which can be empirically tested. It reports research findings that can be explained by the model, and it derives practical suggestions for instructional design. Finally, the chapter discusses limitations of the model and points out directions for further research.