\documentstyle[apalike,html]{article}

\setlength\textheight{9in}
\setlength\textwidth{6.2in}
\setlength\headheight{0in}   \setlength\headsep{0in}
\setlength\topmargin{0pt}
\setlength\footheight{.5in}   \setlength\footskip{.5in}
\setlength\oddsidemargin{.1in}
\setlength\evensidemargin{.1in}
\setlength\parskip{.1in}

\input{defs}

\title{Playpen: Toward an Architecture for Modeling the Development of
Spatial Cognition\thanks{
This report has been prepared in the form of an HTML document, which is
available on the World-Wide Web at
{\tt ftp://ftp.cs.indiana.edu/pub/gasser/Playpen/TR1/tr/tr.html}.
It makes extensive use of color figures, animated applets, and hyperlinks.
There is a compressed Postscript version of the report available by
anonymous ftp 
to {\tt ftp.cs.indiana.edu} at {\tt /pub/gasser/Playpen/TR1.ps.Z}.
This includes the color figures but not the applets and hyperlinks.
Therefore we highly recommend the version on the Web.
}}
\author{$^{1,2}$Michael Gasser and $^{1}$Eliana Colunga\\
$^{1}$Computer Science Department\\
$^{2}$Linguistics Department\\
$^{1}$Cognitive Science Program\\
Indiana University\\
Bloomington, IN 47405, USA\\
{\tt \{gasser,ecolunga\}@cs.indiana.edu}}

\begin{document}

\maketitle

\fignotitle{baby1-small.ps}
\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/baby1-small.gif}

\pagebreak

\begin{abstract}
In this report we argue that the study of the acquisition of
word meaning requires taking seriously non-linguistic cognition, in
particular human vision and the pre-linguistic development of
concepts.
We consider the implications of this claim for the acquisition of
spatial relations, and
we present Playpen, an evolving neural network
architecture for modeling the development of spatial language and
spatial cognition.
Playpen includes modules for high-level vision, the lexicon, and the
conceptual space in which vision and lexicon come together, allowing
for the mutual influence of all three.
This report focuses on the basic building blocks of the network.
Feature binding and object segregation are implemented through the use
of phase angles, and the learning algorithm is a version of Contrastive
Hebbian Learning
\cite{movellan}, adapted for units with phase angles.
We argue that to represent and learn the meanings of relational terms,
the network also requires units which represent micro-relations
explicitly.
In Playpen these take the form of 
\textbf{relation units}, hard-wired clusters of simpler units which
become activated to the extent that they receive inputs from units
representing distinct objects.
\end{abstract}

\section{Introduction}
\label{intro}

How do words get their meanings?
Does this process depend on the details of what the
words are about, that is, on the way things actually are in the
world?  If it does, does it also depend on the
mechanisms within learners which allow them to deal with and
understand the world, that is, with their sensory/perceptual and motor
systems? 
And if this is the case, does the way the world is understood by
learners depend in turn on word meaning and how it is learned?

These are questions that have been around for a long time.  They bear
on issues as fundamental as what language is and what symbolic
cognition in general is.  In this paper we will argue briefly that
much can be gained by assuming that the answer to all three questions
is yes: regularities in the world and the mechanisms of perception and
action matter for language, and language in turn matters for
cognition.  Then we will discuss some methodological consequences of
taking this position, and we will consider what this position means
for a particular semantic domain, that of spatial relations.  Finally,
we will present Playpen, an evolving
connectionist model designed to simulate the development of spatial
cognition and spatial language.  We believe that the fundamental
questions can only be answered by such a model, one which makes
concrete predictions about the behavior that children actually
exhibit.

\section{Modeling Language Acquisition}
\label{modeling}

\subsection{Meaning, Concepts, and Perception in Models of Language
Acquisition}

Where does linguistic meaning fit into cognition?  In
this section we consider three possible positions one could take on
this question and some of the implications of these positions.

All models must deal with the obvious fact that different languages
have different semantic systems, that they divide up the world in
different ways.  One important distinction between models concerns the
way in which language-specific semantics is hooked up to the rest of
the cognitive system in such a way that linguistic behavior is roughly
appropriate to the non-linguistic context.

Probably the most widely-held view is one in which linguistic meaning
is viewed as a symbolic system which maps onto a universal and
symbolic conceptual system.
This conceptual system may be innate or learned
pre-linguistically.  For example, this is the view of two very
influential frameworks, those of \htmlref{Pinker (1994)}{node500pinker94} 
\nocite{pinker:94} and \htmlref{Jackendoff (1992)}{node500jackendoff92} 
\nocite{jackendoff:92}.
We will refer to models that adhere to this view as
\textbf{symbolic} models of linguistic behavior and acquisition.
On the symbolic view, language acquisition involves learning about the
particular language's semantic structures and how they map onto
universal conceptual structure.  Conceptual structure is in turn
related to perception and action in ways which are usually left
unspecified.  An overview of such a perspective is
shown in \hyperref{Figure 1}{Figure~}{}{fig:pinker.eps}.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/pinker.gif}
\fig{\textbf{Symbolic Semantics, Symbolic Concepts}.
Meaning and non-linguistic concepts arise independently of each
other.}{pinker.eps}

If this position holds, then language acquisition can be studied and
modeled without taking perception into account.  The acquisition of
semantics is a symbolic phenomenon, relating one developing symbolic
system, the semantics of the target language, with another, the
existing conceptual system.  Furthermore, since basic conceptual
structure is in place before language is learned, this position does
not allow for any significant influence of linguistic categories on
concepts.  Finally, because semantic development in any language is
driven by the same underlying conceptual system, there should be
strong similarities in the developmental course of the learning of
meaning across diverse languages.

A range of alternatives to this popular view have been set forth in
recent years.  These positions agree in assigning more significance to
the interplay between linguistic and non-linguistic cognition.  We
will refer to them as {\bf grounded} models of linguistic behavior
and/or acquisition.  Grounded models are associated with cognitive
linguists
\cite{lakoff:women,langacker:87}
and with other cognitive scientists who seek to do away with mind-body
distinction in one sense or another
\cite{harnad:grounding,johnson:87,thelen+smith,varelaetal}.

We consider here two possible positions within the space of grounded
models.  The first is that of \htmlref{Regier (1996)}{node500regier96},
\nocite{regier:96}
whose computational model of the acquisition of spatial relation terms
is one of a small number of serious attempts to actually implement the
grounding idea.  In Regier's model, linguistic meaning is learned
directly via perception, and
acquisition can only be studied in the context of a model of the
vision system.  The nature of this system constrains the kinds of
possible meanings that languages can encode and the way in which these
meanings are learned by children.  However, Regier's model makes a
clear division between vision and language and has no obvious place
for spatial concepts.  Presumably the acquisition of spatial terms has
little or nothing to do with spatial reasoning, which in any case is
not under the influence of linguistic categories.  Furthermore, the
model only runs in the production direction; it does not tell us how a
child learns to comprehend spatial terms.  A schematic of Regier's
model is shown in \hyperref{Figure 2}{Figure~}{}{fig:regier.eps}.  
The figure does not assign a place to 
non-linguistic concepts; presumably these would exist in a component
of the system parallel to language.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/regier.gif}
\fig{\textbf{Grounded Semantics}.  Meaning arises directly out
of perception but is kept separate from non-linguistic
concepts.}{regier.eps}

A second, more radical, option within the space of grounded models
also has linguistic meaning grounded in perception.  The difference
here is that there is no distinction between linguistic meaning and
non-linguistic concepts and that the model runs in both
directions, from language to vision as well as from vision to
language.  Particular
meanings/concepts may be learned in three ways: through non-linguistic
perceptual and motoric experience, through a combination of
non-linguistic and linguistic experience, and through linguistic input
alone.  This type of
model allows linguistic categories to influence concepts;
that is, various forms of linguistic relativism
\cite{gumperz+levinson}
are possible.  Finally such a model predicts that the
developmental course of the learning of meaning should depend on the
categories inherent in the language being learned.  A schematic of
this sort of model is shown in \hyperref{Figure 3}{Figure~}{}{fig:mg+ecl.eps}.  
This is the view that we 
favor and the one that is realized in Playpen, the connectionist model
we present \htmlref{later}{playpen} in this report.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/mg+ecl.gif}
\fig{\textbf{Grounded Semantics/Concepts}.  Meaning and concepts are
not distinguished; interactions between language and
perception can occur.}{mg+ecl.eps}

\subsection{Horizontal and Vertical Approaches to Language}

Language is too large a domain to deal with in its entirety, and language
scientists must slice up the problem space in some way or another.
Normally the slices made are \textbf{horizontal}.
A body of research covers some aspect of language, for example, syntax
or the syntax of relative clauses,
or some form of linguistic behavior, for example, syntactic parsing
or the parsing of relative clauses.
The goal is relatively thorough coverage of the behavior.
Contact is often not made with other aspects of language or
linguistic behavior; for example, research on syntax may not make reference
to phonology or pragmatics and research on parsing may not make reference
to production or acquisition.
And contact is even less often made with non-linguistic aspects of
cognition or with the external world.

\textbf{Vertical approaches}, on the other hand, make explicit contact
across different aspects of language or linguistic behavior or across
the boundary between linguistic and non-linguistic.
While such approaches are ``tall,'' they are of necessity also ``thin;''
they can cover only very narrow aspects of language or linguistic
behavior.
Vertical approaches are associated in particular with
cognitive linguistics, with anthropological linguistics, and
with sociolinguistics.
\hyperref{Figure 4}{Figure~}{}{fig:approaches.eps} illustrates horizontal and vertical
approaches to language. 

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/approaches.gif}
\fig{\textbf{Vertical and Horizontal Approaches to
Language}.  Approaches may cover crucial interactions in toy worlds
(vertical) or a single domain in a relatively broad fashion
(horizontal).}{approaches.eps}

While they have the advantage of broad coverage of a domain, horizontal
approaches may miss crucial interactions between domains.
A model such as the one we proposed in the last section, one in
which language and non-linguistic perception exert mutual influence on
one another, obviously requires a vertical methodology.
Language, vision, and non-linguistic concepts must all be taken
seriously.
This is a tall order, enough to daunt even someone who believes in the
sorts of interactions we are suggesting.
Such an approach can succeed only if

\begin{enumerate}

\item the range of linguistic phenomena covered is very narrow

\item there is a body of established results or a coherent
theory to guide the modeling in each of the relevant domains.

\end{enumerate}

We have chosen to focus on the
\textbf{language of spatial relations}
and how it emerges in children.  Besides the cross-linguistic study of
how space is depicted linguistically and the acquisition of spatial
language, our modeling will take us into the vision system, in
particular, the visual representation of relations, and into the
development of concepts of space in children.
We believe that progress in each of these areas has reached the point
where one may attempt to tie the various pieces together.
Our eventual contribution will be an integrated picture of the
development of spatial relations, linguistic and otherwise.
In the following
sections we summarize briefly some relevant facts from these four
areas.
First we discuss the
language of spatial relations and how spatial relations are acquired in
different languages. Then we talk
about the perceptual end, about vision in particular,
and about the acquisition of spatial concepts
independent of language.  Finally we look at the
interactions between the two ends.

\section{Constraints on the Model}

\subsection{Language}
\label{constraints:language}

\subsubsection{The Language of Spatial Relations}
\label{language+space}

Each language provides speakers, hearers, and learners with a finite
set of lexical items and structures to apply to a continuous world, and
it is convenient to view a language as ``slicing up'' the world in a
particular way.
There seem to be both universal and language-specific aspects to the
way this happens.
All languages apparently make a fundamental distinction between nouns
on the one hand and several classes of words on the other, most
centrally, verbs.
It has been argued
\cite{langacker:nouns},
though not uncontroversially, that this distinction
corresponds to a fundamental conceptual distinction between
\textbf{objects} and \textbf{relations}.

The nouns of a language divide the world into categories of objects
and substances.\footnote{
Not all nouns refer to physical objects or substances, of course, but
all of the early nouns learned by children do.}
Verbs, prepositions, and postpositions break things
down in a quite different way from nouns, singling out relations between the
objects which the nouns refer to.
Spatial relations are an important subcategory, and what is striking
here is the relatively small number of discrete spatial relation
categories that each language makes available.
The \textbf{relation term} itself may be a preposition, postposition,
verb, or even a noun inflection; morphological details will not
concern us further.
A complete \textbf{spatial relation expression} includes, in addition
to the relation term itself, two noun phrases, representing the thing
being related (the \textbf{trajector}) and the thing it is being related
to (the \textbf{landmark}).
The choice of trajector and landmark matters:
\textit{the stick is on the block} does not mean the same thing as
\textit{the block is under the stick}.
Trajector seems to correlate with the perceptual figure 
\cite{herskovits:book,langacker:87}.

%Spatial relation expressions are of two types: noun phrases,
%functioning as reference acts
%(e.g., \textit{the stick on the block}),
%and clauses, functioning, among other uses, as assertions
%(e.g., \textit{the stick is on the block}).
%For the purposes of this paper, we will only be concerned with the latter.
%Furthermore, since grammatical categories such as definiteness,
%specificity, and tense will be irrelevant, we will ignore the
%grammatical morphemes in an expression and speak of English spatial
%relation expressions in the form \textit{noun preposition noun}.

Even a cursory examination of the spatial relation expressions in a
subset of languages reveals that the space of possible relations is
sliced up in a variety of ways.
Consider some of the possibilities for encoding relations of
\textsc{contact}, \textsc{support}, and \textsc{containment}
between two objects
\cite{landau:96}.
Four possible arrangements of a trajector (black) and landmark (brown)
are shown \hyperref{Figure 5}{Figure~}{}{fig:LXrelations.eps}.
Spanish uses a single word, \textit{en}, for all of them.
English uses one word, \textit{on}, for the two situations in which
\textsc{containment} does not enter in and another, \textit{in}, for
situations in which the trajector is (at least partially) contained in
the landmark.
German distinguishes two kinds of situations for which English uses
\textit{on}: \textit{auf} when the landmark is under the trajector,
\textit{an} when the trajector is fixed to a vertical surface of the
landmark.
Korean distinguishes two kinds of \textsc{containment} (and
\textsc{contact}) situations, those in which the trajector fits tightly
within the landmark, for which \textit{sok} is used, and those in
which there is loose fit, for which \textit{ahn} is used.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/LXrelations.gif}
\fig{\textbf{Spatial Relations Across Languages}.
Different languages divide up the space of \textsc{contact/support}
relations in different ways.}{LXrelations.eps}

%Another example is concerned with expressions of \textsc{separation}
%and \textsc{joining} in English and Korean. 
%While English makes distinctions based on \textsc{containment}
%(\textit{in} vs. \textit{out}) and \textsc{surface contact/support}
%(\textit{on} vs. \textit{off}),
%Korean makes distinctions based on \textsc{tight/loose fit/attachment}
%(\textit{kkita} vs. a range of other verbs)
%\cite{choi+bowerman}.

But it is not languages which ``slice up'' the world in particular
ways (languages don't actually ``do'' anything); it is people.
In any case our goal is to model individual language learners,
not the entire linguistic communities which embody particular languages.
Descriptions of language and particular languages are useful to us
only insofar as
they give us clues about what people must learn to do in order to
learn language.

\label{ou-motivation}
Linguistic descriptions tell us that language is to a large extent
about objects; thus a major task for
language users and language learners is to find and categorize objects
in the world.
Within the visual-spatial world, they must be able to (1)~segregate
a scene into distinct regions associated with distinct objects,
(2)~cognitively ``bind'' together the features associated with each
distinct object, and (3)~assign these cognitive
objects to the categories represented by the different nouns
of the language.
In \hyperref{Section 4.2.3}{Section~}{}{object-units} 
we discuss a mechanism which satisfies these basic constraints.

\label{ru-motivation}
Linguistics also tells us that all languages have ways of
explicitly encoding relations, so
people must be able to find relations in the world and categorize them
appropriately.
Even if we assume that each scene contains only one salient relation,
they must have the ability to 
(1)~segregate a scene into distinct objects and bind their features
together ((1) and (2) above),
(2)~cognitively bind together the relational features associated
with a given candidate relation,
(3)~assign trajector and landmark status to the related objects, and
(4)~assign the cognitive representation of relations to the categories
represented by the different relation terms of the language.
We believe that these requirements point to an explicit way of
representing relational information.
Furthermore, since languages differ considerably in the sort of
breakdown they make within the space of possible relations,
the human capacity to represent and learn spatial relations must be
a flexible one.
Rather than a set of pre-existing relational categories,
what is called for is a set of relational building blocks from which
the relational categories of different languages can be assembled.
In \hyperref{Section 4.2.4}{Section~}{}{relation-units} we describe 
a representational scheme of this type.

\subsubsection{The Acquisition of Linguistic Spatial Relations}
\label{language+acquisition}

As we have \htmlref{just seen}{language+space}, languages look very different from one another
with respect to space.
We would like to know whether the differences really matter for
the acquisition process. 
Views such as the symbolic position of Pinker and Jackendoff would predict
little effect: since all children start out with the same universal
spatial categories, they should go through roughly the same stages in
acquiring spatial language. 
In particular where a language does not encode one of the universal
categories in a direct way, we would expect over-generalization errors
in which a particular form is applied to the universal category. 
Work by Bowerman and colleagues
\cite{choi+bowerman} on the acquisition of English and other languages
has shown that this is not the case.
Korean children, for example, use no global semantic categories of
\textsc{containment} and \textsc{surface contact/support}, categories
which are not expressed in Korean in a transparent way.
Instead they learn the Korean distinction between \textsc{tight} and
\textsc{loose fit} early on. 
The data seem to support the view that the particular semantics of the
lexicon of the target language has a significant effect on the way the
language is learned.

We think this view is correct.
Thus any model of the acquisition of spatial language must account not
only for
(1)~the developmental path babies follow in learning spatial terms,
but also for
(2)~the interaction
between the particular lexicon of the language being learned and the
way it is learned.  This is an argument for a model of the type shown
in \hyperref{Figure 3}{Figure~}{}{fig:mg+ecl.eps}, one in which 
linguistic meaning and concepts are not clearly distinguished.

\subsection{Non-Language}
\label{constraints:non-language}

Spatial cognition in the child emerges from the convergence of
perceptual and motoric experience, and a complete characterization of
its development certainly requires attention to vision, touch,
proprioception, locomotion, and object manipulation, as well as
spatial language.
We have chosen to start, however, with what is the most
important source of perceptual information about space, at least for
the seeing infant: vision.

\subsubsection{Vision and Imagery}
\label{vision}

Obviously an overview of the human vision system is beyond the scope
of this report.  We will only be concerned here with what is most
directly relevant to spatial relations.  We follow closely
\htmlref{Kosslyn's (1994)}{node500kosslyn94}
\nocite{kosslyn:94}
model of high-level vision and imagery because
(1)~it takes into account the whole range of subsystems that
are involved in high-level vision and (2)~it is meant to
account for ``cognitive graphics'' as well as vision; that is, it also
runs in the concepts-to-vision direction.  The terms we use below are
Kosslyn's.
The components we are concerned with are illustrated in
\hyperref{Figure 6}{Figure~}{}{fig:vision.eps}.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/vision.gif}
\fig{\textbf{Basic Components of the Vision System} (based on Kosslyn,
1994).  Separate modules are responsible for what is seen in the
current Attention Window and where objects are in the current Visual
Buffer.  Vision makes contact with the rest of cognition in the
Associative Memory.}{vision.eps}

It is generally agreed that the vision system divides into a subsystem
responsible for \textbf{What} is in an observed scene and a subsystem
responsible for \textbf{Where} the objects in the scene are.
%\cite{a bunch of people} and mention where these things are in the brain
Among the tasks of the What system is the categorization of objects in
the scene, a process which permits the assignment of noun labels to
the objects.  Similarly, the Where system categorizes relations
between objects in the scene, a process which permits the assignment
of relation terms to the relations in the scene.  Thus both subsystems
are crucial to the task we are interested in.

For our purposes, visual processing begins in a Visual Buffer (VB), a
series of feature-specific maps which have already benefited from edge
detection and region filling.  The VB's task, among others, is to
segregate the scene into regions associated with different objects.

The VB is scanned by an Attention Window (AW), which permits the
system to focus on a single object at a time.  The AW provides the
interface between the VB and the What system, which extracts features
spanning more than a ``pixel'' and ultimately categorizes the contents
of the AW.  The system operates not only in a bottom-up direction,
however; there are top-down influences both on object categorization
and on the placement of the AW.

On the Where side, output from the VB is assigned a 3D coordinate
system, viewer-centered or object-centered (or both), and the
location, size, and orientation of each object in the scene are
extracted.  Later the Where system is responsible for classifying the
relations between objects in the scene.  Relations are of two types,
categorical relations (such as \textsc{containment}) and continuous
relations (such as \textsc{x-centimeters-from}).  As on the What side,
categorization depends on top-down influences as well as on the
strictly visual bottom-up ones.

At its ``top,'' the visual system makes contact with non-visual
cognition in an Associative Memory (AM).  Both the What and Where
systems play a role in the AM and are in turn under its influence when
there are top-down effects on object or relation categorization and
when the system runs in the imagery direction.  It is in this AM that
vision and language come together.

Kosslyn (1994) has also amassed considerable evidence that mental images
share many of the properties of actual percepts.
For example, scanning a mental image takes time proportional to
the distance between imaged objects. 
This evidence suggests that visual mental imagery and visual
perception share mechanisms, that imagery amounts in a sense to
running the vision system in reverse.

For our purposes, then, two points are important:
\begin{enumerate}
\item The vision system has separate modules for handling objects
(What) and for handling relations (Where), and these modules have
access to different kinds of information in the raw input.
\item Imagery makes use of the same basic mechanisms as vision.
\end{enumerate}

\subsubsection{Pre-Linguistic Development of Spatial Relations}
\label{pre-linguistic}

Average children spend about a year in the world before producing
their first
word and it will take them around six months more to start learning words
at a fast pace. Babies do not spend this time in idle contemplation of the
world; they spend it learning about how their own body
works, and more relevant to our current argument, about how the world works.
Very young infants display a knowledge of what happens and what doesn't
happen in the world; they can predict the consequences of actions and be
surprised when their expectations are violated.
 
For example, babies know about how objects behave in support or occlusion
events. Babies as young as 4.5 months realize that objects which are not
supported will fall \cite{needham+baillargeon93} and 8.5-month-old
infants can judge whether an object is 
being sufficiently supported and be surprised when an insufficiently 
supported object fails to fall
\cite{baillargeon+hankosummers90}. 
Young infants also know about the 
impenetrability of objects and the parts of objects that should still be 
visible given the shape of the occluder
\cite{baillargeon92}.
Other results show that this knowledge develops.
For example, very young babies are not 
surprised by seemingly unsupported stable objects if there is an occlusion
event between the habituation and the test
\cite{spelkeetal92a}.
Also, very young babies are not
surprised when unsupported objects fail to fall if there is no motion
involved in the event \cite{spelkeetal92b}.
 
Babies also have the more abstract notion of ``objectness''. Young
infants are
able to use some of the cues adults use to segregate objects, such as 
relative motion and textural cues 
%\cite{spelkeetal93}.
\cite{needham+baillargeon97}.
This knowledge also develops: motion
comes first, then textural information, and finally gestalt properties
\cite{spelkeetal93}. 
Four-month-old infants 
expect objects to retain characteristics such as size and trajectory even
though they are not visible
\cite{baillargeon91a}.

Infants also seem to be able to categorize spatial
relations.
Three- to 4-month-old babies categorize \textsc{left-right}
\cite{behlChadha+eimas95}
and \textsc{on-under} \cite{quinn94} relations,
generalizing over the orientation, the size, and the absolute location
of the objects involved in the relation.
Seven-month-old infants are able to generalize over different
\textit{kinds} of objects
involved in the relation.
Again, this knowledge develops; younger infants are not able
to abstract over different objects involved in the relation, while older
infants can \cite{quinnetal96}.

In sum, before they have learned any words, children seem to be forming
categories that are
useful in representing what the world is like. We think that the learning
that occurs during this period is important in setting the
basis on which linguistic concepts will be formed; hence our model has
a pre-linguistic learning period.
%We also believe the developmental path
%babies follow in the pre-linguistic acquisition of spatial concepts is
%relevant, so our model should be constrained to follow the same path.

\subsection{Interactions Between Language and Non-Linguistic Perception}
\label{interaction}

It seems clear that language and perception have something to do with each 
other. At the most superficial level we know we can describe what we perceive
using words and can also ``imagine'' what is described to us in 
words.  However, we believe language and perception are deeply
interrelated in ways that go beyond these obvious connections and
that these inter-relationships should be taken seriously by any model of
language acquisition.

Consider first the influence of non-linguistic perception on
linguistic behavior and language acquisition.
Obviously what is perceived influences the choice of words used to
describe it, but our perceptual
experience could also directly influence our acquisition of language.
There are at least two possibilities: (1)~The way in which
the world is construed on particular 
occasions may have an impact on how language is learned.
(2)~Specific perceptual mechanisms or categories may be
prerequisites for the acquisition of specific words or structures.

The first sort of relationship can be shown in an
experimental setting by looking at how people generalize nonsense words
to novel situations.  For example, when shown a block on a box while being
told ``the block is acorp the box'' people interpret
\textit{acorp}
to mean \textsc{on}.  In contrast, when shown a stick on a box, people
interpret \textit{acorp} as
\textsc{across} \cite{landau:96}. In the first case, the shape of the
trajector is ignored; in the second it matters.
We suggest that something like this goes on throughout the 
\htmlref{acquisition of language}{language+acquisition}.
When a child hears a word, the world is
generalized according to whatever is perceived in that moment, and
contrast and perceptual saliency affect the way in which
the situation is construed.

But if cognitive linguists such as \htmlref{Langacker (1987)}{node500langacker87} are right,
\nocite{langacker:87}
the influence of perception on the acquisition of language goes beyond
this to the second sort of influence.
For cognitive linguists, grammar is a mapping between form and
function, and they argue that the functional pole of grammatical
patterns is concerned with non-linguistic psychological processes such
as visual scanning, figure-ground segregation, and imagery as well as with
psychological dimensions such as color and depth.  For
example, the grammatical category \textsc{trajector} is defined with
respect to the figure in an observed, recalled, or imagined scene.
The upshot of this position is that the learner of a language needs
access to a relatively direct path from perceptual mechanisms to
language learning mechanisms so that such relationships can be
acquired, and grammatical structures and words can only be learned
once the requisite psychological processes and categories are in
place.  In other 
words, since language is, in a very direct way, about perception,
language acquisition relies on the perceptual capacities of the
learner.
Some evidence for this relationship comes from research showing that
there is a 
correlation between conceptual development and linguistic development
in semantic domains such as space and time
\cite{weistetal:97}.
While correlations do not establish a causal relationship in one
direction or the other,
the most plausible explanation for these results seems to be one in
which language acquisition presupposes conceptual categories, the
sorts of categories that arise out of 
\htmlref{perceptual learning}{pre-linguistic}.

Now consider the influence of language on non-linguistic cognition.
There are three possible ways in which such effects could occur.
(1)~The
wording of a particular utterance could influence the way in which the
state or event that is referred to is conceptualized or remembered.
(2)~The language being learned could become associated with the
contexts in which it occurs, an effect we could look for in bilinguals.
(3)~The regularity implicit in the grammar or lexicon of a
particular language could favor certain patterns of thought on the
part of the speakers of the language.

When we hear a description, we form images in our minds.
As noted in \hyperref{Section 3.2.1}{Section~}{}{vision} above, 
these images resemble visual
percepts and seem to make use of the visual system itself.
Different linguistic descriptions of the same scene may evoke quite
different images:
the noun phrase \textit{half watermelon} is more likely than the noun
\textit{watermelon} to cause subjects to include \textit{seeds} in a
list of features
\cite{wu:95}.
The way a scene is described may also alter our memory of it.
For example,
people who see a green car and then have it
described as ``blue'' are more likely to recognize a more bluish car as
the one they saw before than people who didn't hear it labeled 
\cite{loftus+palmer}, and
people who are asked to label
non-prototypical color chips perform worse
on a recognition task than people who did not label them during study 
\cite{schooler+schooler}.
%A more impressive effect of language-driven categorization on what is
%mainly thought of as a perceptual task comes from experiments in which
%people prove to be better at finding
%Os in a set of numbers than in a set of letters when the Os are
%called ``oh'' and better at finding them in a set of letters when they
%are called ``zero''
%\cite{jonides+gleitman}.
%Since the two percepts are identical, it is clearly the top-down
%influence of language that is at work here.

A deeper relationship between language and thought emerges when we
examine the influence of the specific language on how the world is
perceived.
One possibility is that a language may tie
its speakers to particular contexts.
Experiments with bilinguals have shown that their two languages evoke
different contexts.
Chinese-English 
bilinguals were presented with descriptions of individuals and then 
asked whether the individuals described were likely to have certain
behaviors.
Subjects addressed in Chinese extrapolated using Chinese stereotypes
and subjects addressed in English used English stereotypes
\cite{hoffmanetal}. 

A more controversial possible relation between language and
non-linguistic cognition concerns the effects of the regularities
inherent in particular languages, what is usually known as
\textbf{linguistic relativism}.
In its strongest form, this position, associated most strongly with
the ideas of \htmlref{Benjamin Lee Whorf (1956)}{node500whorf},
\nocite{whorf}
holds that categories in the grammars and lexicons of particular
languages have a direct impact on the thought patterns of speakers.
As we have seen in
\hyperref{Section 3.1.1}{Section~}{}{language+space}, different 
languages ``slice up'' the world in different ways.
If a  language explicitly codes for a certain distinction, making such
a distinction might become relatively easy for speakers of that
language.
There has been relatively little systematic investigation of
relativism
\cite{lucy:96},
so, despite some intriguing evidence in favor of an influence of
language on perception and thought \cite{lucy:92},
it is still premature to assume that such an influence is pervasive.
Our position is that computational modeling may shed light on the
possibility of the language-to-perception/thought relationship in a way
that has not been possible before.
Given the evidence, we believe that models must remain open to the
possibility of such a relationship by maintaining the
language-to-concepts-to-vision path in the architecture.
Excluding this path precludes any sort of relativism.

Thus we find at least some evidence for all of the following sorts of
influences:
\begin{enumerate}
\item perception $\rightarrow$ attention $\rightarrow$ language acquisition
\item perception $\rightarrow$ construal $\rightarrow$ language acquisition
\item wording $\rightarrow$ attention/memory
\item choice of language $\rightarrow$ attention/memory
\item linguistic regularity $\rightarrow$ concepts $\rightarrow$ perception
\end{enumerate}

To summarize, there seem to be a number of ways in which linguistic and
non-linguistic perception interact.  Memory,
attention, and categorization are influenced by both linguistic and
non-linguistic input, and memory, attention, and
categorization, in turn, influence both language and perception.
These interactions have important effects on the acquisition of
language and belong in a model of acquisition.
Models of the type shown in 
\hyperref{Figure 3}{Figure~}{}{fig:mg+ecl.eps} incorporate
these interactions.
Playpen, the model we describe in the next section, is such a model.

\section{Playpen}
\label{playpen}
Playpen is an evolving model of the development of spatial cognition
and spatial language. 
We call it ``Playpen'' because it is meant to deal only with a simple
world of blocks, sticks, and containers and because the regularities
in the physical external playpen that the model lives in should be
incorporated into the network itself as it learns: in a sense the
network will ``have'' whatever it has discovered about the playpen in
its connection weights.

\subsection{Playpen's Task}
\label{task}

Eventually we would like the network to be the brains of a robot
which exists in a playpen-like world of blocks and other simple
objects.
In the short run, we must simulate the world.
This requires an account of what regularities there are in the world,
including linguistic regularities.
Even when Playpen is embedded in a robot, a theory of the
development of spatial cognition must include an account of what
is out there to be learned.
For the purposes of this paper, we will remain vague in this regard;
details will come out when we consider particular linguistic
relations, as we will do in the next report on Playpen.
We do assume, however, that the linguistic input to the child
makes a fundamental distinction between expressions for things
and expressions for relations.
This fact imposes constraints on the network: it must have the
capacity to represent \htmlref{objects}{ou-motivation} as collections 
of features and \htmlref{relations}{ru-motivation}
as pairs of objects (which are collections of features).

The world that infants are exposed to includes language, as well as
other sorts of stimuli, from the start.
But because infants have not yet figured out the phonology of the
target language, linguistic input is probably completely irrelevant
for the learning of spatial concepts.
Once children are capable of recognizing particular words, normally
early in the second year, they are in a position to learn the spatial
concepts associated with words.
Thus it is convenient to divide children's task into two phases,
a \htmlref{\textbf{pre-linguistic} phase}{pre-linguistic} in which 
they learn about space as they
observe and manipulate the things around them, and a
\textbf{linguistic} phase in which they also receive linguistic input,
input within which they are capable of distinguishing particular words.
In this latter phase, non-linguistic learning about space continues,
but it is supplemented by the co-occurrence of words and
phrases with particular scenes.

\subsection{Playpen's Architecture and Behavior}

Playpen's architecture and behavior are
constrained both by the nature of its task and what is
known about how language and vision operate.

\begin{enumerate}

\item As discussed in \hyperref{Sections 3.2.1}{Sections~}{}{vision} 
and \hyperref{3.3}{~}{}{interaction}
above, processing is interactive and
bi-directional.  Visual input can yield linguistic output, and
linguistic input, along with some visual context, can yield visual
output in the form of expectations or imagery.

\item In the \htmlref{pre-linguistic phase}{pre-linguistic}, the
system receives visual inputs
with no explicit labels, and learning consists of somehow extracting
regularity from these inputs.
In the linguistic phase, some visual inputs are accompanied by
linguistic inputs as well.
In one sense this just means that the input patterns are more complex;
in another it means that there are now new independent grounds for
dividing space up into particular categories.

\item Language is explicit about certain categories, in particular
\htmlref{objects}{ou-motivation} and \htmlref{relations}{ru-motivation}.
Learning language requires that the network have the capacity to
represent these categories explicitly.

\item A good deal is known about the human \htmlref{vision system}{vision}.
While we believe there are large gaps that need to be filled in, the
model should be in general agreement with the vision facts. 

\item A good deal is known about 
\htmlref{language development}{language+acquisition} and 
\htmlref{concept development}{pre-linguistic}. 
Playpen is meant to model the behavior of babies, and
the most important constraints are those concerned with development:
What precedes what?  What is hard or easy?

\end{enumerate}

We have paid most attention to the first four constraints.
The fifth is likely to play a greater role as we model the
development of particular relations.

\subsubsection{The Overall Architecture}

While we are currently focusing on vision and language and their
interaction, the model should eventually bring together input from
other perceptual domains --- proprioception and touch --- and from
movement --- locomotion, the movement of a limb, and the manipulation
of objects with a hand --- as well. 
The overall architecture we envision is shown in
\hyperref{Figure 7}{Figure~}{}{fig:playpen.eps}.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/playpen.gif}
\fig{\textbf{Playpen Architecture.}  Spatial concepts emerge out of the
interplay of non-linguistic
perceptual input, linguistic input, and action.
Portions we focus on are highlighted in boldface or
with thick lines.}{playpen.eps}

The visual component of Playpen is based loosely on 
\htmlref{Kosslyn's model}{fig:vision.eps} of vision and imagery 
\cite{kosslyn:94}. 
\hyperref{Figure 8}{Figure~}{}{fig:playpen-zoom.eps} shows how vision 
and language interact in the model.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/playpen-zoom.gif}
\fig{\textbf{Playpen Architecture: Vision and
Language}.  Visual objects and nouns are tied together on the What
side of the system.  Visual and linguistic relations are tied together
on the Where side.
Most of what is shown has not been implemented; the
regions we have focused on are highlighted in boldface or
with thick lines.}
{playpen-zoom.eps}

The overall organization of the network is such that higher layers
roughly preserve the spatial relations within lower layers;
higher representations are more abstract than lower ones.
The input visual layers, the Visual Buffer, are topological maps.
The Visual Buffer performs bottom-up object segregation.
The Attention Window is a mainly stimulus-driven mechanism
which zooms in on a part of scene in the Visual Buffer corresponding
roughly to a putative object.
The Attention Window passes on a region in the Visual Buffer to the
What system, which categorizes the object which is in
the Attention Window, adding a representation of the object to
an \textbf{Object Short-Term Memory},
a component not found in Kosslyn's model.
The Where system receives the entire scene from the Visual Buffer.
The segregation of the scene into regions associated with the
different objects is preserved, but 
lower-level layers in this system are responsible for assigning
perspective to the scene, both object-centered and viewer-centered,
and higher-level layers extract salient dimensions such as position along
the vertical dimension and object size. 
The representations provide input to the \textbf{Spatial Relation
Concepts} layer, where the system categorizes spatial relations
pre-linguistically.
The Object Short-Term Memory and the Spatial Relation Concept layer
interact so that it is possible for a relation together with its
arguments to be represented, as is required, for example, for the
meaning of a spatial relation expression.

The Language layer has units for two types of words, nouns and
relation terms (prepositions in English). 
The What side of the visual system connects to the nouns, permitting
labeling of objects and, in the comprehension direction, the
understanding of nouns as visual patterns within the What system and
the Visual Buffer.
The Where side of the visual system connects
to the relation terms, permitting labeling of relations and, in the
comprehension direction, together with the What side, the
understanding of relation expressions as 
visual patterns in the Where system and the Visual Buffer.

\subsubsection{Processing Units}
\label{units}

The network is of the generalized Hopfield type: connections between
units are symmetric, and units repeatedly update until the network
settles.
Each unit has an associated activation function; for most units this
is the familiar interactive activation rule
\cite{mcclelland+rumelhart:iac}:

If $h_{i}^{t} > 0$,
\begin{equation}
\Delta a_{i}^{t} = h_{i}^{t} (a^{max}_{i} - (a_{i}^{t-1} -
	 D_{i} a_{i}^{t-1}))
\end{equation}

Else,
\[\Delta a_{i}^{t} = h_{i}^{t} ((a_{i}^{t-1} - D_{i} a_{i}^{t-1}) -
a^{min}_{i})\]

\noindent
where $a_{i}^{t}$ is the activation of unit $i$ at time $t$;
$h_{i}^{t}$ is the input to $i$ at time $t$; and $a^{max}_{i}$,
$a^{min}_{i}$, and $D_{i}$ are the maximum activation, minimum
activation, and decay rate associated with $i$.
All units in the network currently have maximum activations of 1 and
minimum activations of 0.

As we noted in \hyperref{Section 3.1.1}{Section~}{}{language+space} 
above, language makes
explicit reference to objects 
and relations, and a model of the acquisition and processing of
language requires a means of representing conceptual objects and
relations, as well as the associations between lexical/grammatical patterns
and conceptual objects and relations.
The network is made up
of two kinds of units: \textbf{Object Units} and \textbf{Relation
Units}, which serve these two purposes.

\subsubsection{Object Units}
\label{object-units}

Object units (OUs) are units with a 
\textbf{relative phase angle} in addition to an activation.
As in a number of other recent models
\cite{hummel+biederman,shastri+ajjanagadde,spornsetal:89},
synchronization functions to bind together the features of distinct
objects.  Units with the same phase angle
are part of the same object, and units with different phase angles
belong to different objects.

The connection between each pair of OUs has not only a weight but
also an associated
\textbf{coupling function}, a function of the difference in
phase angles of the two units.
The coupling function must be symmetric about 0, and its
derivative must be anti-symmetric about 0; see the 
\hyperref{Appendix}{Appendix}{}{appendix} for why
these constraints must hold.
Both the activation and the
phase angle of an OU are potentially modified each time a unit is 
updated, and both depend on the coupling function on the weights into
the unit.
The input $h$ and change in phase angle $\Delta \varphi$ to an OU $i$
are given by
 
\begin{equation}
h_{i} = 
\sum_{j=1}^{n}
a_{j} \cdot w_{ij} \cdot \Phi_{ij} (\varphi_{i} - \varphi_{j})
\end{equation}
 
\begin{equation}
\Delta \varphi_{i} =
\frac{\pi}{\sum_{j=1}^{n}}
\sum_{j=1}^{n} a_{j} \cdot w_{ij} \cdot
\Phi_{ij}^{\prime} (\varphi_{i} - \varphi_{j}),
\end{equation}
\noindent
where $n$ is number of units in the network, $a_{j}$ is the activation
of unit $j$, $w_{ij}$ is the weight
connecting units $i$ and $j$, and $\Phi_{ij}$ is the coupling function
associated with units $i$ and $j$.
A stable state of the network is, then, a state in 
which neither activations nor phase angles are changing.

The most common coupling function used in the network is
$.5 + .5 \cos x$.
For positive weights, the system consisting of the two units with this
coupling function has an attractor at the state where the units are in
phase and a repeller at the state where they are out of phase.
The two units excite each other at all phase angle differences except
$\pi$.
For negative weights,
there is an attractor at the out-of-phase state and a repeller at the
in-phase state, and the units inhibit each other except when they are
out of phase.

Demo~\thedemo\ illustrates the behavior of a small
network of OUs.

\demo{Object Units}

The units in Playpen's lexicon representing nouns are also OUs.
Learning the meaning of a noun would involve creating positive
connections between the noun unit and the associated non-linguistic
visual/spatial feature units.
These connections would tend to cause the connected units to align
their phase angles, so that
in the comprehension or production of a phrase the word is ``bound''
to its meaning.
\hyperref{Figure 9}{Figure~}{}{fig:OU-words.eps} illustrates the 
relationship between
noun OUs and the associated non-linguistic ``semantic'' OUs.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/OU-words.gif}
\fig{\textbf{OUs in the Representation of Noun Meaning}.
Nouns and visual objects are both represented by OUs; during
production or comprehension synchronized phase angles bind together
nouns and their internal ``referents''.
Unit activation is represented by brightness, and phase angles are
indicated by the blue triangles.  The dashed green arrows represent
trainable connections which have developed positive
weights.  Only a minimum number of units is shown.}{OU-words.eps}

\subsubsection{Relation Units}
\label{relation-units}

\paragraph{The Problem}

We have seen how OUs permit the network to distinguish clusters of
features from one another and how positive connections between noun
OUs and non-linguistic ``semantic'' OUs can implement the binding that
is part of understanding and producing nouns.

Now consider what would be required for relations, both relation terms
and non-linguistic relations.
A binary, non-reflexive relation relates two distinct objects, for
example, a stick and a block in
\textsc{support(block, stick)}.
To implement a relation in a distributed connectionist network, we
first need to make two modifications to the standard view from
predicate logic.
First,
rather than treat an expression like this as a predicate with a truth
value, we will treat it as a set of correlations or inferences.
In this sense, \textsc{support} in the example means loosely that
stick features and 
block features in particular relative positions correlate with one
another.
Viewed in terms of inference, the relation takes the form of a process
of pattern completion, for example,
\begin{enumerate}
\item Given a supported stick, infer that the supporter is a block.
\item Given a block and stick in a support relationship, infer that
the block is the supporter.
\end{enumerate}
Note that the inferences are not absolute; whether they actually hold
depends on a number of other factors, for example, whether the stick
in question has a flat surface.
In the network implementation, whether a particular relation holds
would depend on the combined input coming into the relevant units and
therefore on the entire network of other relational inferences in the
system.
Note also that from this connectionist perspective, there is no
distinction between a relation
between individual objects and a generalized relation over types of
objects.

A second modification to the standard view of relations is to treat
them as non-atomic.
A distributed implementation of a relation involves multiple
\textbf{micro-relations}, each relating a pair of features, one
belonging to each of the two objects.
A micro-relation represents a highly specific micro-inference.
Thus knowledge about \textsc{support} is made up of micro-relations
specific to particular relative locations in some abstract
representational space.

Within the Playpen framework described so far, we could represent
a micro-relation as a negative connection between two OUs.
If both units are activated, this connection causes them to repel each
other.
With different phase angles, they would then correspond to parts of
different objects.
Thus the micro-inference represented by such a connection would be:
\textit{given whatever features are associated with unit A and
whatever features are associated with unit B, the A features and the B
features belong to two different objects}.
Recall, however, that a negative connection represents inhibition as
well as repulsion.
Thus the desired inference only holds to the extent that the two units
are both activated, for example, when both have their activations
clamped.
When only the phase angles can change, the two units will tend to end
up out-of-phase, indicating distinct objects.
However, when both activation and phase angle are permitted to change,
the negative connection can also result in one unit's inhibiting the
other, the precise outcome depending on the initial state of the two
units, the coupling function associated with the connection, and of
course other weights and inputs into the units.
%Demo~\thedemo\ 
%illustrates these possibilities for a simple network of two
%OUs.
%\demo{Simple Relations}

However, this way of representing relations gets us nowhere when it
comes to correlations between relations.
For example, we might want to represent the following
inference:
\textit{if A supports B, then A is probably larger than B}.
This involves an association between a \textsc{support} relation and a
\textsc{bigger-than} relation, thus in the network minimally four OUs,
two for each relation.
A further example is the association required to specify the meaning
of a relation term:
\textit{if ``A is on B'', then some object $A'$ is on another
object $B'$}.
This involves an association between a linguistic \textit{on}
relation and a non-linguistic \textsc{on} relation.
Any set of connections we set up among the four units necessary to two
relations fails to capture the relationship between the relations that
we want.
Consider the connections shown in
\hyperref{Figure 10}{Figure~}{}{fig:OU-relation.eps}.
While the individual connections do seem to represent the phase
relationships we want for this example, if we consider the connections
separately, we see that they miss crucial conditions.
Thus the positive connection between the \textsc{high} unit in the
\textsc{location} pair and the \textsc{small} unit in the
\textsc{relative size} pair indicates that objects that are high tend
to be small.
But what we would like to convey is the more complex fact that objects
that are higher than other objects tend to be smaller than the other
objects.
The relationship we want to represent requires that the
micro-relations be treated as units, and the simple network shown in
\hyperref{Figure 10}{Figure~}{}{fig:OU-relation.eps} does not permit this.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/OU-relation.gif}
\fig{\textbf{Attempt at Representing Relation-Relation
Correlations.}  Two units in each of two different layers on the Where
side of the network are shown.
Green connections have positive
weights, red connections negative weights.
This network offers no way to relate the location and size relations
to one another directly.}{OU-relation.eps}

Note that a conjunctive unit connected to both of the OUs in a
particular micro-relation does not solve the problem either.
While we could arrange for such a unit to turn on to the extent that
the two input units are on (using positive connections which affect
activation but not phase angle),
the unit would fail to represent the relation because it has no way of
holding onto the two phase angles of the inputs: the identity of the
two objects in the relation is lost.

Apparently relations between relations require explicit units
representing micro-relations and taking
OUs and other relation units as inputs to their two role ``arms.'' 
The relation units (RUs) should

\begin{enumerate}
\item Become activated to the extent that both input (groups of) units
are activated and the two (groups of) units are out-of-phase with each
other.
\item Preserve the phase angles of the inputs, passing them on to
other RUs.
\item Associate with other RUs via a mapping of the arms which
preserves the appropriate phase angle relationships and also reflects
the strength of the correlation between the two micro-relations.
\end{enumerate}

\paragraph{The Solution}

RUs, which are the major innovation of the Playpen architecture,
are used to represent relational information; they are
``about'' two different objects.
Each RU is made up of a cluster of five units 
hard-wired in such a way that the unit as a whole is activated to the
extent that it is receiving input from two distinct objects.
\hyperref{Figure 11}{Figure~}{}{fig:RU.eps} shows a pair of RUs along with four OUs
illustrating the connectivity within the RUs and between them and
other units of both types.
An RU has two \textbf{interfaces}, each consisting of a pair of OUs:
one to handle interaction with other RUs, the other to handle
interaction with OUs.
The two interfaces are connected in such a way that the corresponding
arms tend to be in-phase and the opposing arms tend to be
out-of-phase.
The RU is considered activated when all four of its interface units
are activated.
Each RU also has a simple unit with no phase angle and negative
connections to all four of the OUs.
This \textbf{bias unit} has a resting activation of 1.0 and turns off
only when its input falls below a threshold.
The bias unit functions to prevent the OUs on one interface from
turning on those on the other interface unless they are both
sufficiently activated.
Without the bias unit, ``one-armed'' relations, those in which one arm
only is activated on each interface, would be possible.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/RU.gif}
\fig{\textbf{Relation Units.}  Two RUs are shown, each surrounded by a
magenta border.  Both are in their active state: all four arms are
maximally activated and the bias units are inhibited.
The two RUs are connected to each other on their RU interfaces, and
each RU is also connected to two OUs on its OU interface.}{RU.eps}

Connectivity between RUs is also constrained, though the weights
themselves are trainable.
There are four connections joining each pair of connected RUs on their
RU interfaces, but only two distinct weights, one for the connections
joining corresponding arms, the other for the connections joining
opposing arms.
The coupling function on these connections is
$.5 + .5 \cos x$.
These connections implement the four possible relationships that can
exist between micro-relations:

\begin{enumerate}

\item The two micro-relations have no effect on each other.
In this case both weights are 0.

\item The two micro-relations correlate negatively with one another;
that is, the presence of one leads one to expect the absence of the
other. In this case both weights are negative.

\item The two micro-relations correlate positively with one another,
and corresponding arms are bound to the same object.
In this case the weight for the corresponding arms is positive and
that for the opposing arms 0.

\item The two micro-relations correlate positively with one another,
and opposing arms are bound to the same object.
In this case the weight for the corresponding arms is 0 and
that for the opposing arms positive.

\end{enumerate}

In the resting state of an RU, the four OUs have activations of 0 and
the bias unit an activation of 1.
When an RU is activated on one or the other or both of its interfaces
in such a way that the two OUs on an interface are highly activated
and out-of-phase, the bias unit is inhibited, and the OUs on the
interface (if not already activated) become active and take on the
phase angles of the corresponding OUs in the other interface.
When the RU is completely activated, the four OUs have
activations of 1 and the bias unit an activation of 0.
Demo \thedemo\ illustrates a simple network of two RUs,
one of which is activated by input from a single OU.

\demo{Relation Units}

Now consider how RUs would permit us to represent the meaning of a
spatial relation term.
The word itself has an associated RU with a trajector and a landmark
arm on both interfaces.
The RU interface of this RU connects to
appropriate RUs in the Spatial Relation Concepts layer, which in turn
connect to OUs specifying the locations associated with features of
the two related objects.
The nouns representing the trajector and landmark of the relation must
be in phase with the corresponding arms of the relation term RU.
We are far from a complete account of how this takes place in either
production or comprehension.
For now, we simply assume that a trajector OU implements this binding
process.
\hyperref{Figure 12}{Figure~}{}{fig:RU-words.eps} illustrates these 
relationships.

\htmladdimg{/l/ftp/pub/gasser/Playpen/TR1/Gifs/RU-words.gif}
\fig{\textbf{Relation Units in the Representation of Word
Meaning.} Only one of many possibly relevant units is shown in the
Spatial Relation Concepts and Location layers.  The RU interfaces of
the RUs appear on top, the OU interfaces on the bottom.  Only the
positive connections between the two RUs are shown; there would be also be
connections with weight 0 joining the opposing arms of the two RU
interfaces. The trajector OU implements the phase relationship between
the trajector arm of the word RU and the corresponding noun
OU.}{RU-words.eps}

\subsubsection{Sequential Time}
\label{time}

Though our concern is with {\em static\/} relations, we cannot
ignore movement.
In order to categorize relations, children must be able to segregate
the scene into distinct objects.
However, before seven months, they are generally unable to segregate
static configurations of objects
\cite{spelke:90}, and
\htmlref{Thelen and Smith (1994)}{node500thelensmith}
\nocite{thelen+smith}
have proposed that they may learn to
use static properties to perform object segregation by observing
objects move and then come to rest in space.

If we are to deal with movement, we must first have a means of dealing
with time.
In a generalized Hopfield network, we face the problem that settling
itself requires time.
The response of the network to temporal inputs or the network's
generation of temporal output must take place at a time scale beyond
that necessary for settling.
Most approaches to time in neural networks incorporate some sort of
short-term memory, a means by which units respond not only to the
current state of the network but also to a limited record of its
previous states.
Within the generalized Hopfield framework, this can be accomplished
through the inclusion of 
\textbf{delays} on connections, an idea due originally to
\htmlref{Kleinfeld (1986)}{node500kleinfeld86}.
\nocite{kleinfeld:86}
Connecting any two units in the network there may be any number of
connections, each with its own delay.
Input to a unit along a connection is a function of the activation and
phase angle of the destination unit at the time before the delay.
\hyperref{Equation 4}{Equation~}{}{delay} shows this relationship.

\begin{equation}
h_{i}^{t} = 
\sum_{j=1}^{n} \sum_{k=0}^{m}
a_{j}^{t-k} \cdot w_{ij}^{k} \cdot \Phi_{ij}^{k} (\varphi_{i}^{t} -
\varphi_{j}^{t-k})
\label{delay}
\end{equation}

\noindent
where input, activation, and phase angle are a function of time ($t$) and
$m$ is the maximum delay.

Demo \thedemo\ shows the behavior a simple network with two hard-wired
connections between each pair of units, one with no delay and one with
a delay of one ``primitive time step.''

\demo{Delay Connections}

\subsubsection{Learning}
\label{learning}

The network is trained using a variant of Contrastive Hebbian Learning
(CHL) \cite{movellan}, modified to accommodate unsupervised learning
(auto-association) and phase angles.
In Contrastive Hebbian Learning, weight updates take place in
positive (Hebbian) and negative (anti-Hebbian) phases.
During the positive phase, the input and output units are clamped to
training patterns, and the network is then allowed to settle.
Weight updates during this phase are proportional to the product
of the activations of the units on either end of the connection.
During the negative phase, only the input units are clamped, and the
network is then allowed to settle.
Weight updates during this phase are proportional to the negative of
the product of the activations on either end of the connection.
For OUs, which have phase angles, the coupling function applied to the
phase angle difference also enters in.
The net weight update on the connection joining units $i$ and $j$
following the presentation of a training pattern is

\begin{equation}
\Delta w_{ij} \propto
\breve{a}_{i}^{+} \cdot \breve{a}_{j}^{+} \cdot
\Phi_{ij}(\breve{\varphi}_{i}^{+} - \breve{\varphi}_{j}^{+}) -
\breve{a}_{i}^{-} \cdot \breve{a}_{j}^{-} \cdot
\Phi_{ij}(\breve{\varphi}_{i}^{-} - \breve{\varphi}_{j}^{-}),
\end{equation}

\noindent
where ($\breve{\ }$) over a symbol refers to that quantity when the
network has stabilized and the $+$ and $-$ superscripts refer to the
positive and negative phases of learning.
When the network generates the appropriate output for each training
input, the changes from the positive and negative phases cancel each
other out, and the weights no longer change.
See the Appendix for the derivation of the the CHL rule for OUs.

CHL was originally described for hetero-associative learning.
For auto-associative learning, there is no distinction between input
and output units, so we must decide which units are clamped and which
not clamped during the negative phase.
One way to proceed is to randomly select input/output units to clamp
during the negative phase, and it is this approach that we have
followed with Playpen.

Demo \thedemo\ illustrates the two phases of learning in a
simple network of OUs.

\demo{Learning}

\section{Conclusions and Future Work}

In the first part of this report we tried to convince you that in order to
understand where the meanings of words come from, it makes sense to
look outside as well as within language, especially at human vision and
at pre-linguistic conceptual development.
But how is one to take on the complexity of these diverse domains,
each of which occupies an entire research community?
We believe that these research communities have given us much to go
on, that we can take enough off the shelf to get us started, and that
by slicing our domain of interest very thin, specifically by confining
ourselves to simple static spatial relations, it is possible to build
an insightful model of the development of word meaning.

In the second part of the report we outlined the beginning of the
Playpen project, whose goal is such a model.
We described a simple neural network architecture based on features of
the human vision system which allows for the emergence of spatial
concepts from the interaction of vision and language.
We emphasized three basic building blocks which we believe are
required: object units, relation units, and delay connections.

In a subsequent technical report (available in September 1997), we
describe how Playpen models the 
relative difficulty of the acquisition of the words
\textit{on}, \textit{under}, \textit{left}, and \textit{right}.

Future work includes

\begin{enumerate}

\item implementation of other components of the architecture,
especially those in the What system

\item application of the model to other specific relations, especially
\textsc{in} and \textsc{behind}.

\item taking the visual world seriously, eventually using input from
a camera.

\end{enumerate}

\pagebreak

\appendix

\section{Mathematical Details of the Model}
\label{appendix}

In this section we show how Contrastive Hebbian Learning (CHL)
\cite{movellan} needs
to be modified to accommodate units with relative phase angles.
We follow the derivation in Movellan closely.\footnote{
We ignore the possibility of delay connections, but they do not affect
the derivation of the learning rule.}

Movellan defines a continuous Hopfield Energy function

\begin{equation}
F = E + S
\label{energy}
\end{equation}

\noindent
where $E$ reflects the constraints imposed by the weights in the
network and $S$ the tendency to drive the activations to a resting
value.
For our network $S$ is the same as for a network with no phase angles:

\begin{equation}
S = \sum_{i=1}^{n} \int_{rest_{i}}^{a_{i}} f_{i}^{-1} (a) da
\label{penalty}
\end{equation}

\noindent
where $n$ is the number of units in the network, $a_{i}$ is
the activation of unit $i$, $f_{i}$ is the
activation function for unit $i$, and $rest_{i} = f(0)$.

\noindent
However, $E$ becomes

\begin{equation}
E = -\frac{1}{2} \sum_{i=1}^{n}\sum_{j=1}^{n} a_{i} \cdot w_{ij}
   \cdot a_{j} \cdot \Phi_{ij}(\varphi_{i} - \varphi_{j})
\end{equation}

\noindent
where $w_{ij}$ is the weight connecting
units $i$ and $j$ and 
$\Phi_{ij}$ is the coupling function associated with units $i$ and
$j$.
In what follows we will abbreviate $\Phi_{ij}(\varphi_{i} -
\varphi_{j})$
as $\Phi_{j \rightarrow i}$.

The coupling function must be differentiable and satisfy the following:
\begin{equation}
\Phi_{i \rightarrow j} = \Phi_{j \rightarrow i}
\label{cpl_symmetry}
\end{equation}
\begin{equation}
\Phi_{i \rightarrow j}' = - \Phi_{j \rightarrow i}'
\label{cplprime_antisymmetry}
\end{equation}

When the network is stable, the inverse of the activation function for
each unit is equal to the input into that unit:

\begin{equation}
f_{i}^{-1} (\breve{a}_{i}) = \breve{h}_{i} =
\sum_{j=1}^{n} \breve{a}_{j} w_{ij} \breve{\Phi}_{j \rightarrow i}
\label{equilibrium_input}
\end{equation}

\noindent
where ($\breve{\ }$) represents equilibrium and $h_{i}$ is the input
to unit $i$.
Furthermore, when the network is stable, the phase angle of each unit
no longer changes:

\begin{equation}
\Delta \breve{\varphi}_{i} =
\frac{\pi}{\sum_{j=1}^{n} w_{ij}}
\sum_{j=1}^{n} \breve{a}_{j} w_{ij} \breve{\Phi}_{j \rightarrow i}' = 0
\label{equilibrium_phase}
\end{equation}

Movellan defines the contrastive function $J$ as

\begin{equation}
J = \breve{F}^{(+)} - \breve{F}^{(-)}
\end{equation}

\noindent
and shows that the CHL rule minimizes $J$.
We follow his derivation for the case where units have phase angles.

The energy of the network $E$ at equilibrium is

\begin{equation}
\breve{E} = -\frac{1}{2} \sum_{i=1}^{n}\sum_{j=1}^{n}
\breve{a}_{i} w_{ij} \breve{a}_{j} \breve{\Phi}_{j \rightarrow i}
\end{equation}
\noindent
Extracting the terms with a $w_{ij}$ term,

\begin{equation}
\breve{E} =
-\frac{1}{2} \left(
2\breve{a}_{i}w_{ij}\breve{a}_{j}\breve{\Phi}_{j \rightarrow i}
+ \sum_{k=1}^{n}
\sum_{\stackrel{l=1}{\scriptscriptstyle k,l\neq i,j;k,l\neq j,i}}^{n}
 \breve{a}_{k} w_{kl} \breve{a}_{l} \breve{\Phi}_{l \rightarrow k}
  \right)
\end{equation}
\noindent
Differentiating with respect to a single weight $w_{ij}$ and
considering that $w_{ij}$ is the only weight depending on $w_{ij}$,

\begin{eqnarray}
\frac{\partial\breve{E}}{\partial w_{ij}} & = &
-\frac{1}{2}
\left[
 2\breve{a}_{i}\breve{a}_{j} 
\breve{\Phi}_{j \rightarrow i} \nonumber + 
2 w_{ij} \breve{a}_{i} \breve{\Phi}_{j \rightarrow i}
\frac{\partial \breve{a}_{j}}{\partial w_{ij}} +
2 w_{ij} \breve{a}_{j}
\breve{\Phi}_{j \rightarrow i}
\frac{\partial \breve{a}_{i}}{\partial w_{ij}} +
\right. \\
& & \breve{a}_{i} w_{ij} \breve{a}_{j}
\left(
\breve{\Phi}_{j \rightarrow i}'
 \left( \frac{\partial\breve{\varphi}_{i}}{\partial w_{ij}}
- \frac{\partial\breve{\varphi}_{j}}{\partial w_{ij}}\right) +
\breve{\Phi}_{i \rightarrow j}'
\left( \frac{\partial\breve{\varphi}_{j}}{\partial w_{ij}}
- \frac{\partial\breve{\varphi}_{i}}{\partial w_{ij}}\right)
\right) \nonumber + \\
& & 
\left. \sum_{k=1}^{n} \sum_{\stackrel{l=1}
{\scriptscriptstyle k,l\neq i,j;k,l\neq j,i}}^{n} w_{kl} 
\left(
\breve{a}_{k} \breve{\Phi}_{l \rightarrow k}
\frac{\partial \breve{a}_{l}}{\partial w_{ij}} +
\breve{a}_{l} \breve{\Phi}_{l \rightarrow k}
\frac{\partial \breve{a}_{k}}{\partial w_{ij}} +
\breve{a}_{k} \breve{a}_{l} \breve{\Phi}_{l \rightarrow k}'
\left( \frac{\partial\breve{\varphi}_{k}}{\partial w_{ij}}
- \frac{\partial\breve{\varphi}_{l}}{\partial w_{ij}}\right) \right)
\right]
\label{energy_hairy}
\end{eqnarray}

\noindent
From Equation \ref{cplprime_antisymmetry}, we have

\begin{eqnarray}
& \breve{a}_{i} w_{ij} \breve{a}_{j}
\left(
\breve{\Phi}_{j \rightarrow i}'
\left( \frac{\partial\breve{\varphi}_{i}}{\partial w_{ij}}
- \frac{\partial\breve{\varphi}_{j}}{\partial w_{ij}} \right) +
\breve{\Phi}_{i \rightarrow j}'
\left( \frac{\partial\breve{\varphi}_{j}}{\partial w_{ij}}
- \frac{\partial\breve{\varphi}_{i}}{\partial w_{ij}} \right)
\right) \\
& = 2 w_{ij} \breve{a}_{i} \breve{a}_{j}
\breve{\Phi}_{j \rightarrow i}'
\frac{\partial\breve{\varphi_{i}}}{\partial w_{ij}} +
2 w_{ij} \breve{a_{i}} \breve{a_{j}}
\breve{\Phi}_{i \rightarrow j}'
\frac{\partial\breve{\varphi_{j}}}{\partial w_{ij}} \nonumber
\end{eqnarray}

\noindent
and

\begin{equation}
\sum_{k=1}^{n} \sum_{\stackrel{l=1}
{\scriptscriptstyle k,l\neq i,j;k,l\neq j,i}}^{n} w_{kl} 
\left( \breve{\Phi}_{l \rightarrow k}'
\left( \frac{\partial\breve{\varphi_{k}}}{\partial w_{ij}}
- \frac{\partial\breve{\varphi_{l}}}{\partial w_{ij}}\right) \right) =
2 \sum_{k=1}^{n} \sum_{\stackrel{l=1}
{\scriptscriptstyle k,l\neq i,j;k,l\neq j,i}}^{n}
w_{kl} \breve{a_{k}} \breve{a_{l}}
\breve{\Phi}_{l \rightarrow k}'
\frac{\partial\breve{\varphi_{k}}}{\partial w_{ij}}
\end{equation}

\noindent
Substituting these into Equation \ref{energy_hairy},

\begin{equation}
\frac{\partial\breve{E}}{\partial w_{ij}} =
-\frac{1}{2} \left(
2\breve{a}_{i}\breve{a}_{j}\breve{\Phi}_{j \rightarrow i} +
2 \sum_{k=1}^{n}\frac{\partial\breve{a}_{k}}{\partial w_{ij}}
\sum_{l=1}^{n}w_{kl}\breve{a_{l}}
\breve{\Phi}_{l \rightarrow k} +
2 \sum_{k=1}^{n} \breve{a_{k}}
\frac{\partial\breve{\varphi_{k}}}{\partial w_{ij}}
\sum_{l=1}^{n}w_{kl}\breve{a_{l}}
\breve{\Phi}_{l \rightarrow k}'
\right)
\end{equation}

\noindent
From \ref{equilibrium_input} and \ref{equilibrium_phase}, we have the
following for the case where $i \neq j$.  Since there are no
self-recurrent connections in our network, we need only consider this
case.

\begin{equation}
\frac{\partial \breve{E}}{\partial w_{ij}} =
-\breve{a}_{i}\breve{a}_{j}\breve{\Phi}_{j \rightarrow i} -
 \sum_{k=1}^{n} \breve{h}_{k} \left(\frac{\partial
 \breve{a}_{k}}{\partial w_{ij}}\right) -
\sum_{k=1}^{n} \Delta\breve{\varphi_{k}} \breve{a}_{k}
\frac{\partial\breve{\varphi}_{k}}{\partial w_{ij}}
\end{equation}

\noindent
From \ref{equilibrium_phase}, the last term is 0, and we have
\begin{equation}
\frac{\partial \breve{E}}{\partial w_{ij}} =
-\breve{a}_{i}\breve{a}_{j}\breve{\Phi}_{j \rightarrow i} -
 \sum_{k=1}^{n} \breve{h}_{k} \left(\frac{\partial
 \breve{a}_{k}}{\partial w_{ij}}\right)
\end{equation}

\noindent
From Equation \ref{penalty},

\begin{equation}
\frac{\partial \breve{S}}{\partial w_{ij}} =
\sum_{k=1}^{n}f_{k}^{-1}(\breve{a}_{k})
\frac{\partial \breve{a}}{\partial w_{ij}}
\end{equation}

\noindent
and from Equation \ref{equilibrium_input}, we have

\begin{equation}
\frac{\partial \breve{F}}{\partial w_{ij}} =
-\breve{a}_{i}\breve{a}_{j}\breve{\Phi}_{j \rightarrow i}
\end{equation}

\noindent
making

\begin{equation}
\frac{\partial \breve{J}}{\partial w_{ij}}
  \propto \breve{a}_{i}^{(+)} \breve{a}_{j}^{(+)}
\breve{\Phi}_{j \rightarrow i}^{(+)} -
\breve{a}_{i}^{(-)} \breve{a}_{j}^{(-)}
\breve{\Phi}_{j \rightarrow i}^{(-)}
\end{equation}

\noindent
which shows that the modified CHL rule

\begin{equation}
\Delta w_{ij} \propto \breve{a}_{i}^{(+)} \breve{a}_{j}^{(+)}
\breve{\Phi}_{j \rightarrow i}^{(+)} -
\breve{a}_{i}^{(-)} \breve{a}_{j}^{(-)}
\breve{\Phi}_{j \rightarrow i}^{(-)}
\end{equation}

\noindent
descends in the $J$ function.

\bibliographystyle{/nfs/salsa/u/gasser/Tex/myapa}
\bibliography{/nfs/salsa/u/gasser/Bib/cx,%
/nfs/salsa/u/gasser/Bib/psych,/nfs/salsa/u/gasser/Bib/ling,ecl}
\end{document}
