It seems clear that language and perception have something to do with each other. At the most superficial level we know we can describe what we perceive using words and can also ``imagine'' what is described to us in words. However, we believe language and perception are deeply interrelated in ways that go beyond these obvious connections and that these inter-relationships should be taken seriously by any model of language acquisition.
Consider first the influence of non-linguistic perception on linguistic behavior and language acquisition. Obviously what is perceived influences the choice of words used to describe it, but our perceptual experience could also directly influence our acquisition of language. There are at least two possibilities: (1) The way in which the world is construed on particular occasions may have an impact on how language is learned. (2) Specific perceptual mechanisms or categories may be prerequisites for the acquisition of specific words or structures.
The first sort of relationship can be shown in an experimental setting by looking at how people generalize nonsense words to novel situations. For example, when shown a block on a box while being told ``the block is acorp the box'' people interpret acorp to mean ON. In contrast, when shown a stick on a box, people interpret acorp as ACROSS [Landau, 1996]. In the first case, the shape of the trajector is ignored; in the second it matters. We suggest that something like this goes on throughout the acquisition of language. When a child hears a word, the world is generalized according to whatever is perceived in that moment, and contrast and perceptual saliency affect the way in which the situation is construed.
But if cognitive linguists such as Langacker (1987) are right,
the influence of perception on the acquisition of language goes beyond this to the second sort of influence. For cognitive linguists, grammar is a mapping between form and function, and they argue that the functional pole of grammatical patterns is concerned with non-linguistic psychological processes such as visual scanning, figure-ground segregation, and imagery as well as with psychological dimensions such as color and depth. For example, the grammatical category TRAJECTOR is defined with respect to the figure in an observed, recalled, or imagined scene. The upshot of this position is that the learner of a language needs access to a relatively direct path from perceptual mechanisms to language learning mechanisms so that such relationships can be acquired, and grammatical structures and words can only be learned once the requisite psychological processes and categories are in place. In other words, since language is, in a very direct way, about perception, language acquisition relies on the perceptual capacities of the learner. Some evidence for this relationship comes from research showing that there is a correlation between conceptual development and linguistic development in semantic domains such as space and time [Weist et al., 1997]. While correlations do not establish a causal relationship in one direction or the other, the most plausible explanation for these results seems to be one in which language acquisition presupposes conceptual categories, the sorts of categories that arise out of perceptual learning.
Now consider the influence of language on non-linguistic cognition. There are three possible ways in which such effects could occur. (1) The wording of a particular utterance could influence the way in which the state or event that is referred to is conceptualized or remembered. (2) The language being learned could become associated with the contexts in which it occurs, an effect we could look for in bilinguals. (3) The regularity implicit in the grammar or lexicon of a particular language could favor certain patterns of thought on the part of the speakers of the language.
When we hear a description, we form images in our minds. As noted in Section 3.2.1 above, these images resemble visual percepts and seem to make use of the visual system itself. Different linguistic descriptions of the same scene may evoke quite different images: the noun phrase half watermelon is more likely than the noun watermelon to cause subjects to include seeds in a list of features [Wu, 1995]. The way a scene is described may also alter our memory of it. For example, people who see a green car and then have it described as ``blue'' are more likely to recognize a more bluish car as the one they saw before than people who didn't hear it labeled [Loftus and Palmer, 1974], and people who are asked to label non-prototypical color chips perform worse on a recognition task than people who did not label them during study [Schooler and Engstler-Schooler, 1990].
A deeper relationship between language and thought emerges when we examine the influence of the specific language on how the world is perceived. One possibility is that a language may tie its speakers to particular contexts. Experiments with bilinguals have shown that their two languages evoke different contexts. Chinese-English bilinguals were presented with descriptions of individuals and then asked whether the individuals described were likely to have certain behaviors. Subjects addressed in Chinese extrapolated using Chinese stereotypes and subjects addressed in English used English stereotypes [Hoffman et al., 1986].
A more controversial possible relation between language and non-linguistic cognition concerns the effects of the regularities inherent in particular languages, what is usually known as linguistic relativism. In its strongest form, this position, associated most strongly with the ideas of Benjamin Lee Whorf (1956),
holds that categories in the grammars and lexicons of particular languages have a direct impact on the thought patterns of speakers. As we have seen in Section 3.1.1, different languages ``slice up'' the world in different ways. If a language explicitly codes for a certain distinction, making such a distinction might become relatively easy for speakers of that language. There has been relatively little systematic investigation of relativism [Lucy, 1996], so, despite some intriguing evidence in favor of an influence of language on perception and thought [Lucy, 1992], it is still premature to assume that such an influence is pervasive. Our position is that computational modeling may shed light on the possibility of the language-to-perception/thought relationship in a way that has not been possible before. Given the evidence, we believe that models must remain open to the possibility of such a relationship by maintaining the language-to-concepts-to-vision path in the architecture. Excluding this path precludes any sort of relativism.
Thus we find at least some evidence for all of the following sorts of influences:
attention language acquisition construal language acquisition attention/memory attention/memory concepts perception
To summarize, there seem to be a number of ways in which linguistic and non-linguistic perception interact. Memory, attention, and categorization are influenced by both linguistic and non-linguistic input, and memory, attention, and categorization, in turn, influence both language and perception. These interactions have important effects on the acquisition of language and belong in a model of acquisition. Models of the type shown in Figure 3 incorporate these interactions. Playpen, the model we describe in the next section, is such a model.