\subsections{Generation of referring expressions}

The generation of referring expressions has been a topic of
significant interest within the field of computational
linguistics. The general parameters of the task are clear: the main
question in this research field is what
linguistic expression to use in order to refer to an entity so that it
is informative enough for the speaker to pick out the referent (among
possibly other distractor entities), and at the same time it does not 
include unnecessary information.  As the entrainment literature would
suggest, the proper form of reference to an entity is likely to be
{\em negotiated} with the other speaker rather than to be produced
according to  a deterministic algorithm. The algorithms for generation of
referring expressions have focused on first reference (the linguistic
expression used in a first mention of an entity in the discourse) , but a dialog
system would need to generate appropriate subsequent references as
well. Their form would depend on the previous dialog. 

Recently, \newsite{Dale06} compared the productions of several
existing algorithms for generation of referring expressions with real
human productions. There were a number of mismatches between the
automatically generated and human descriptions. 1) Peope included more
information than was necessary to uniquely disambiguate the referent
2) No current algorithm could generate any of the relational
descriptions produced by people. ``the orange drawer above the blue
drawer'',  while the relational algorithm would produce descriptions
like ``the drawer above the drawer above the drawer above the pink
drawer''. 3) People produced varied descriptions of the same objects
while all of the generation algorithms are detrministics and always produce
the same reference. This fact need not be considered a problem if we
model the attention of the speaker---the salience of different
properties can change as a result of the interaction.

%
%1. Full brevity algorithm: almost selects the most distinguishing
%property available 
%
%2. Incremental algorithm: preference ordering over the properties 
%-- each entity is characterized by a list of attribute value pairs
%   (what would these be in the travel domain?)

Some other recent research also found out that the existing algorithms
for generation of referring expressions do not capture aspects of
human preference. For example in their experiments Kramer and Thuene
\cite{} have confirmed that people prefer to omit properties in
   subsequent references and they prefer a uniquely identifiable
   entity to be referenced using a pronoun rather than a full
   noun phrase. First references are tentative and once they have been
   established, only the core part of the description can be used.
Moreover, traditional generation algorithms ( most notably the incremental
algorithm) assume that there is a preferred order
   in which people select properties (color>shape>size). In the
   algorithm proposed by Siddhatrthan and Copsake \cite{} a more reasonable
   assumption is adopted, that is that speakers pick out attributes
   that are distinctive {\em in context}. 
Finally, in task oriented dialogues, the users are more likely to adopt the
   terminology of the expert, but in citical situations when there is
   a problem in the communication, the expert can be willing to adopt
   the user's terminology \cite{}.  

A further issue with referring expressions is their acoustic
   realization. Referring expressions are often realized by a long
   noun phrase. Some words within the noun phrase would be made more
   prominent than others and the decision about which parts of the
   reference should be prominent are not trivial
   \cite{sproat,juliaShimei}. In fact in a recent study \cite{aruba}
   it has been confirmed that long noun phrases are most problematic
   for automated prominence prediction systems and many of the system
   errors occur within complex NP contexts. The dynamic salience of
   distinguishing properties can certainly contribute to better
   predict the appropriate prosodic realization of referring expressions.

The work that we propose to do on lexical entrainment will help
clarify the ways in which current generation algorithms are
lacking. The problems we will address and empirically verify include:
1) A unified framework for reference generation including both first
and subsequent mention. Little previous work has been done on
subsequent mention generation.
2) The dialog context specifies what properties and distinctions the
two interlocutors have been attending to. A model of joint attention
can help determine which are the salient properties to be used in
generating referring expressions. This model will also be able to
capture and explain the fact that different referring expressions can
be used for the same entity: in different context, different
distinctions will be activated or salient and thus different
expressions will be warranted. 
3) The salience model will be integrated in a model for prominence
prediction. 

According to Zwaan and Radvansky, reference to main individuals under
discussion is a key dimensions encoded in situation models, alongside
primitives such as space, time, causality, intentionality.

Other work suggests that specific dimensions of situation models can
be aligned. With respect to the spatial dimension, Schober (1993)
found that interlocutors tended to adopt the same reference frame as
each other. When interlocutors face each other, terms like on the left
are ambiguous depending on whether the speaker takes what we can call
an egocentric or an allocentric reference frame. Schober found that
if, for instance, A said on the left meaning on A?s left (i.e., an
egocentric reference frame), then B would subsequently describe
similar locations as on the right (also taking an egocentric frame of
reference). 

There is some evidence for alignment of situation models in
comprehension. Garrod and Anderson (1987) found that players in the
maze game would query descriptions from an interlocutor that did not
match their own previous descriptions (see section 4 below). Recently,
Brown-Schmitt, Campana, and Tanenhaus (in press) have provided direct
and striking evidence for alignment in comprehension. Previous work
has shown that eye movements during scene perception are a strong
indication of current attention, and that they can be used to index
the rapid integration of linguistic and contextual information during
comprehension (Chambers, Tanenhaus, Eberhard, Filip, & Carlson, 2002;
Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1995). Brown-Schmitt
et al. monitored eye movements during unscripted dialogue, and found
that the entities considered by the listener directly reflected the
entities being considered by the speaker at that point. For instance,
if the speaker used a referring expression that was formally ambiguous
but which the speaker used to refer to a specific entity (and hence
regarded as disambiguated), the listener also looked at that
entity. Hence, whatever factors were constraining the speaker?s
situation model were also constraining the listener?s situation
model. 

--- This point below about syntactic priming is also importantant for
the generation of referring expressions. Most algorithms would
``specialize'' in a production of the referring expression stricktly
either employing premodifiers or relational attributes expressed in a
prepositional phrase. The studies cited below show that the dialog
contex would change the preference for one over the other.
  
Levelt and Kelter (1982) found that speakers tended to reply to ``What
time do you close?'' or ``At what time do you close'' (in Dutch) with
a congruent answer (e.g., ``Five o?clock'' or ``At five
o?clock''). This alignment may be syntactic (repetition of phrasal
categories) or lexical (repetition of at). Branigan et al. (2000)
found clear evidence for syntactic alignment in dialogue. Participants
took it in turns to describe pictures to each other (and to find the
appropriate picture in an array). One speaker was actually a
confederate of the experimenter and produced scripted responses, such
as ``the cowboy offering the banana to the robber'' or ``the cowboy
offering the robber the banana.'' The syntactic structure of the
confederate?s description strongly influenced the syntactic structure
of the experimental subject?s description. Their work extends
``syntactic priming'' work to dialogue: Bock (1986b) showed that
speakers tended to repeat syntactic form under circumstances in which
alternative non-syntactic explanations could be excluded (Bock, 1989;
Bock & Loebell, 1990; Bock, Loebell, & Morey, 1992; Hartsuiker &
Westenberg, 2000; Pickering & Branigan, 1998; Potter & Lombardi, 1998;
cf. Smith & Wheeldon, 2001, and see Pickering & Branigan, 1999, for a
review). 
%%% -*- LaTeX -*-
%%% $Id: description.tex,v 1.36 2005/04/29 23:12:08 martin Exp $
\documentclass[11pt]{report}

%%AG: I added these
\usepackage{amsmath,epsfig}

%%\pdfoutput=0  %% asks pdflatex to generate DVI output instead of PDF

\ifx\pdfoutput\undefined
  \relax
\else
  \pdfoptionpdfminorversion=2
  \pdfcompresslevel=9
\fi

\usepackage{type1cm}
\usepackage[activate]{pdfcprot}

%% modified ot1ppl.fd, scaled Palatino by 5%
\DeclareFontFamily{OT1}{ppl}{}
\DeclareFontShape{OT1}{ppl}{m}{n} {<-> [1.05] pplr7t}{}
\DeclareFontShape{OT1}{ppl}{m}{sc}{<-> [1.05] pplrc7t}{}
\DeclareFontShape{OT1}{ppl}{m}{sl}{<-> [1.05] pplro7t}{}
\DeclareFontShape{OT1}{ppl}{m}{it}{<-> [1.05] pplri7t}{}

\DeclareFontShape{OT1}{ppl}{b}{n} {<-> [1.05] pplb7t}{}
\DeclareFontShape{OT1}{ppl}{b}{sc}{<-> [1.05] pplbc7t}{}
\DeclareFontShape{OT1}{ppl}{b}{sl}{<-> [1.05] pplbo7t}{}
\DeclareFontShape{OT1}{ppl}{b}{it}{<-> [1.05] pplbi7t}{}

\DeclareFontShape{OT1}{ppl}{bx}{n} {<->ssub * ppl/b/n}{}
\DeclareFontShape{OT1}{ppl}{bx}{sc}{<->ssub * ppl/b/sc}{}
\DeclareFontShape{OT1}{ppl}{bx}{sl}{<->ssub * ppl/b/sl}{}
\DeclareFontShape{OT1}{ppl}{bx}{it}{<->ssub * ppl/b/it}{}

\usepackage{mathpazo}

\usepackage{geometry}
\geometry{margin=2.5 true cm, nohead, footskip=20bp, includefoot}

%\renewcommand{\baselinestretch}{1.05}
\parindent=1em
\hyphenpenalty=0
\widowpenalty=0
\clubpenalty=0

\usepackage{graphicx}
\graphicspath{{./}}

\usepackage[usenames,dvipsnames]{color}
\definecolor{Black}{rgb}{0,0,0}
%%\ifx\useColor\relax
\ifx\useColor\relax
  \definecolor{Linkcolor}{rgb}{0.7,0.0,0.0}
  \definecolor{Citecolor}{rgb}{0.0,0.5,0.0}
  \definecolor{Filecolor}{rgb}{0.0,0.0,0.5}
\else
  \definecolor{Linkcolor}{rgb}{0,0,0}
  \definecolor{Citecolor}{rgb}{0,0,0}
  \definecolor{Filecolor}{rgb}{0,0,0}
\fi

\usepackage[numbers]{natbib}
\bibliographystyle{abbrvnat}
%%\newcommand{\citeg}[1]{\citeauthor{#1}'s \citeyearpar{#1}}
\renewcommand{\bibname}{References Cited}

\usepackage[pdfpagelabels]{hyperref}
\hypersetup{
  breaklinks=true,
  plainpages=false,
  colorlinks=true,
  anchorcolor=Black,
  linkcolor=Linkcolor,
  menucolor=Linkcolor,
  pagecolor=Linkcolor,
  citecolor=Citecolor,
  filecolor=Filecolor,
  urlcolor=Filecolor,
  bookmarksopen=true,
  bookmarksnumbered=true,
  pdftitle={Collaborative Research: Speaking more like you: Acoustic, 
        prosodic and lexical entrainment in {response generation for?} 
                Spoken Dialogue Systems},
  pdfauthor={Hirschberg et al.},
%
%  pdfsubject={IIS - Collaborative Systems; IIS - Advanced Learning Technologies},
%  pdfkeywords={prosody, language training, spoken language systems},
  %%pdfpagemode=None,
  pdfstartview=FitH
}

\usepackage{hypcap}

\usepackage{rcs}
\setlength{\fboxsep}{8pt}
\newcommand{\CVSinfo}{\smallskip\framebox[\linewidth]{File:
    \RCSRCSfile{}, Revision: \RCSRevision{}, Date: \RCSDate{}
    \RCSTime\,\textsc{utc}}\bigskip}

\usepackage{comment}
\specialcomment{meta}{\medskip\hrule\medskip}{\hspace*{1pt}\hrule\medskip}
\specialcomment{CVS}{}{\CVSinfo}
\excludecomment{meta}
\excludecomment{CVS}

\usepackage{paralist}

\usepackage{ragged2e}

%% Page and section numbering

\renewcommand{\thechapter}{\Alph{chapter}}
\renewcommand{\thepage}{\Alph{chapter}--\arabic{page}}

\setcounter{secnumdepth}{4}  %% number headings down to subsubsections
\setcounter{tocdepth}{4}     %% and put them in the toc

\makeatletter
\renewcommand{\chapter}{%
  \if@openright\cleardoublepage\else\clearpage\fi%
  \global\@topnum\z@%
  \@afterindentfalse%
  \phantomsection%
  \setcounter{page}{1}%
  \refstepcounter{chapter}%
  %%\addtocontents{lof}{\protect\addvspace{18\p@}}%
  %%\addtocontents{lot}{\protect\addvspace{18\p@}}%
  \secdef\MYchapter\MYsChapter}

\newcommand{\MYchapter}[2][?]{%
  \addcontentsline{toc}{chapter}{\protect\numberline{\thechapter}#1}%
  \vspace*{0pt}%
  {\centering\sffamily\bfseries\huge\thechapter\quad #2\par}%
  \vspace*{4ex}%
  }
\newcommand{\MYsChapter}[1]{%
  \addcontentsline{toc}{chapter}{\protect\numberline{\thechapter}#1}%
  \vspace*{0pt}%
  {\centering\sffamily\bfseries\huge\thechapter\quad #1\par}%
  \vspace*{4ex}%
  }
\makeatother

\usepackage[sf,bf,small,compact]{titlesec}

% \titleformat{\section}[wrap]
% {\normalfont\sffamily\bfseries\fontsize{11pt}{14pt}\selectfont\filright}
% {\thesection}{.5em}{}
% \titlespacing{\section}{12pc}{1.5ex plus .1ex minus .2ex}{1pc}

\titleformat{\subsection}[runin]
{\normalfont\sffamily\bfseries}
{\thesubsection}{.5em}{}

%% local macros

\newcommand{\ChapRef}[1]{\hyperref[{chap:#1}]{Chapter~\ref*{chap:#1}}}
\newcommand{\EqRef}[1]{\hyperref[{eq:#1}]{(\ref*{eq:#1})}}
\newcommand{\FigRef}[1]{\hyperref[{fig:#1}]{Figure~\ref*{fig:#1}}}
\newcommand{\SecRef}[1]{\hyperref[{sec:#1}]{\S\ref*{sec:#1}}}
\newcommand{\PageLabel}[2]{\hypertarget{p:#1}{\label{p:#1}#2}}
\newcommand{\PageRef}[1]{\hyperlink{p:#1}{page~\pageref*{p:#1}}}

\let\Acronym=\textsc
\let\Concept=\emph

\newcommand{\YN}{\emph{yes\slash no}}

\newcommand{\FIXME}[1]{\textbf{\textcolor{red}{#1}}}

\clubpenalty=9900
\widowpenalty=9900

%%AG: redefine the 'quotation' environment (used in pilots.tex)
\renewenvironment{quotation}
               {\list{}{
					\topsep 1mm
					\parsep 0mm 
					\itemindent 0mm
					\leftmargin \parindent
			   		\rightmargin \parindent
				}
				\fontfamily{phv}\selectfont
                \item\relax}
               {\endlist}

\begin{document}

\setcounter{chapter}{4}


\input{pilots} 

\clearpage
\RaggedRight
\hyphenpenalty=9999

%\bibliography{refs,refs-previous,/home/julia/bib/refs}
\bibliography{refs,refs-previous,refs-julia}
%%AG: /home/julia/bib/refs.bib has a few errors. I copied it to refs-julia.bib
%%    and corrected them.

\end{document}
\section{Research Plan}
\label{sec:plan}


In Section \ref{sec:pilots} we described some preliminary
investigations on potential areas of entrainment in the Columbia Games
Corpus and the Switchboard Corpus.  We have found indications that
there is lexical entrainment in both corpora and also
acoustic/prosodic entrainment in the Columbia Games Corpus, as
discussed in Section \ref{sec:pilots}.  We have also seen indications
in both corpora that lexical entrainment is accompanied by a number of
different disfluency and hedging behaviors, as speakers negotiate a
conceptual pact among themselves.  In this section we outline our
plans to study these phenomena in both corpora.  Our proposed research
focuses on entrainment in the lexical, acoustic, prosodic and
discourse domains, to pursue the questions raised in Section
\ref{sec:pilots}.  In Section \ref{sec:eval} we detail our planned
approach to building, refining and testing models for the entrainment
types we describe below.

\subsection{Lexical Entrainment: Referring Expressions and Prominence}

As noted in Section \ref{sec:previous}, the presence of lexical
entrainment in human-human communication has been well documented
in the psycholinguistic literature, but current algorithms for the
generation of referring expressions (GRE) do not even attempt to
incorporate a model of entrainment into the generation process.  In
fact, little work has been done on the generation of subsequent
mentions at all.  The proposed work will lead to a computational model
of lexical entrainment that we plan to incorporate into an improved
GRE algorithm, leading to better automatic generation of text.

Recent studies have shown that GRE algorithms are not able to mimic
human productions \cite{Dale06}.  However, suggestions for
the improvement of such algorithms have primarily been based on
preplanned text, rather than spoken dialogue.  For example, Krahmer
and Theune \cite{krahmer00efficient} found in preference experiments
that subjects reading pairs of sentences preferred the omission of
previously mentioned properties when a discourse entity was mentioned
in both sentences; they also preferred that a uniquely identifiable
entity be mentioned using a pronoun, rather than a full noun phrase.
Krahmer and Theune conclude that referring expressions are tentative;
once they have been established, only the core part of the description
should be used.  However, as noted in Section \ref{sec:previous},
\cite{Brennan96a} found that, in conversational data, speakers often
continue to use longer descriptions for discourse entities than would
be necessary to uniquely identify the referent.  This finding would suggest
a different approach then for the generation of referring expressions
in dialogue.

Moreover, traditional generation algorithms (most notably Dale and
Reiter's \TechTerm{incremental algorithm}) \cite{dale95computational} claim
that there is a preferred order in which people order mentioned
properties (size-shape-color, e.g. ``\LingMat{the big round brown X}''). In
Siddharthan and Copestake's \cite{Sidh04} GRE algorithm, it is assumed
that there is no fixed hierarchy of attribute types, but that speakers
pick out attributes that are distinctive \Emphasis{in context}.  Neither approach
allows for the influence of Brennan and Clark's \cite{Brennan96a} conceptual pacts.

%%JH: ani put in previous please
%Finally, in task-oriented dialogues,
%the users are more likely to adopt the terminology of the expert, but
%in citical situations when there is a problem in the communication,
%the expert can be willing to adopt the user's terminology \cite{}.

Referring expressions in dialogue may also exhibit entrainment in
their acoustic and pro\-sod\-ic realization.  Typically in non-pronominal
subsequent reference to discourse entities in monologue or dialogue,
some or all of these previously uttered (given) elements within the
NP will be made less prominent, or \TechTerm{deaccented}.  Predicting
which items in the NP will be deaccented has long been the subject of
study, particularly for text-to-speech systems
\cite{hirschberg95c,Pan00,aruba}.  However, the possibility that in
dialogue such speaker decisions might themselves be influenced by the
productions of the conversational partner has not yet been examined.

%it
%has been confirmed that long noun phrases are most problematic for
%automated prominence prediction systems and many of the system errors
%occur within complex NP contexts. The dynamic salience of
%distinguishing properties can certainly contribute to better predict
%the appropriate prosodic realization of referring expressions.

Our proposed research on lexical entrainment will address the
deficiencies of current generation algorithms in both the lexical
choices and acoustic/prosodic realization of referring expressions in
dialogue.  From a statistical analysis of speaker's generation of
referring expressions in the Games and Switchboard corpora, we will
develop a unified framework for reference generation including both
first and subsequent mention in dialogue, based on an analysis of the
orthographic, prosodic and given/new annotations we have for the
Games and Switchboard corpora.  In these corpora we will identify
patterns of human behavior with respect to the form and content of
subsequent mentions and compare patterns exhibited by each speaker
with his conversational partner(s) to examine the extent to which
lexical and acoustic/prosodic entrainment explains speakers' decisions
on referring expression. For the Games corpus we also have information
on the visual context in which discourse entities are discussed, so we
can test how a model of mutual salience may explain collaborative
decisions on referring expressions.  Since all speakers saw the same
set of objects, but some saw them in the same order and some in
different orders, we should also be able to investigate how different
referring expressions are used for the same entity, in different
contexts and by different speakers, We will integrate this mutual
salience model into procedures for automatic prosody prediction.

\subsection{Acoustic/Prosodic Entrainment}

As noted in Section \ref{sec:previous}, there is some evidence in the
experimental and descriptive literature that speakers adapt to their
conversational partners, changing their own acoustic/prosodic
parameters, such as speaking rate, pitch range and intensity/loudness,
to come closer to those of their conversational partners.  However,
%For example people speak differently when they address children or
%foreigners, or when they speak in a noisy environment.
to date there has been little corpus-based quantitative investigation of
these phenomena in spontaneous human-human dialogues.  

In Section \ref{sec:pilots} we describe some simple experiments that
show evidence of possible entrainment in speaking rate and pitch
variation in the Games Corpus.  We will pursue these investigations in
detail and also study potential entrainment of simple binary
prominence decisions (i.e. Do speakers tend to adapt to their
partner's accent rate?), phrasing behavior, variation in pitch range,
intensity/loudness, speaking rate and duration of pauses.  We will
study these phenomena in both the Games and Switchboard corpora to
determine which of these basic acoustic/prosodic parameters are
subject to adaptation.

We will also investigate the adaptation that may occur in dialogue
speakers' choice of tonal prosodic features such as choice of pitch
contour, pitch accent and phrase endings in the Games Corpus; as
noted in Section \ref{sec:corpus}, this corpus is currently being
labeled in the ToBI scheme.  Certain intonational contours are used to
express particular pragmatic goals, but the same contour can have more
than one pragmatic meaning \cite{Hirschberg02}. There is also evidence
that usage patterns for particular contours are speaker-dependent
\cite{us-labphon9}.  It is possible that speakers may adapt to other
speakers' style of intonational variation in order to express similar
pragmatic purposes, to communicate pragmatic information more
successfully.  Alternatively, intonational entrainment may arise as
more of a priming effect, with speakers entraining upon their
partner's propensity to employ one contour or another, or one type of
pitch accent or phrasal ending, without regard to subtle pragmatic
distinctions.  We will be able to investigate this too in the Games
Corpus, which is currently being labeled for the form and function of
questions, to see whether questions with the same form and function
are realized with different contours by different speakers and whether
entrainment occurs in contour use over a conversation.

%%AN I just made up the above, we might need an example or something
%%more specific, but I am not sure if you guys will be interested in
%%including that, so will hold off on further elaboration for the time being.
  
%hyperarticulation???
%other aspects of timing?

\subsection{Discourse Structuring Entrainment}

Dialogue is rich in other phenomena for which entrainment to one's
conversational partner may be investigated.  In particular, explicit
discourse markers such as cue phrases, backchannels and
signals for turn-taking co-ordination are all important structuring
devices which make spontaneous communication possible.  For this
reason it is important for successful communication that the speakers
achieve a mutual agreement on the use of discourse structuring
expressions.
%% The below is from Deborah Tannens book ``You just don't
%% understand''. Again, I am not sure if there will be interest in
%% keeping this paragraph.
%%JH: hmm the reference is from tannen 86 not tannen 90, which is you
%%just don't understand....
Turn-taking behavior, in particular, has been shown to be both
speaker-dependent and dependent upon cultural background
\cite{Tannen86}.  Beattie's \cite{Beattie82} study of the turn-taking
behaviors of Margaret Thatcher and James Callaghan, for example,
demonstrates significant differences in propensity to interrupt and
ability to hold the floor when another speaker barges in, which
he attributes to differences in behaviors associated with yielding a
turn.  We will investigate whether speakers adapt to their partner's
turn-taking and turn-yielding behaviors over the course of a dialogue
session in the Games Corpus, which has been labeled for turn-taking
behaviors using a Beattie-inspired model.

Similarly, we will address the use of cue phrases, backchannels and
hedges, as well as the generation of laughter and audible breaths
across speakers. Do speakers entrain to another speaker's propensity
to use particular cue phrases over others, such as \LingMat{okay} instead of
\LingMat{yeah} or \LingMat{right}?  Do they tend to adopt their partner's
backchanneling style, either in choice of lexical item or in latency
before backchanneling?  Does one speaker adopt another's hedging
behavior --- become more or less cautious in assertions --- when the
latter's strategy differs from his or her own in terms of lexical
choice or frequency of hedges?  Does a speaker's use and style of
laughter differ when conversing with different conversational
partners?  Does a speaker's use of audible breaths, e.g. as a
turn-taking signal vary when interacting with different speakers?
While a subset of cue phrases, all backchannels, laughter and audible
breaths are already labeled for much of the Games Corpus, the study of
hedging behavior will require additional annotation.

\subsection{Entrainment and Spoken Dialogue Systems}

%%JH: this may sound too optimistic
Our ultimate goal in examining entrainment in the Games and
Switchboard corpora is to incorporate our findings into a fuller model
of entrainment for SDSs.
%If we do indeed find evidence of lexical entrainment, and patterns of
%behavior that characterize entrainment in the use of referring
%expressions, SDS should be able to adopt similar
%strategies in generating both the content and the prosodic form of
%referring expressions.  If we find evidence of acoustic and prosodic
%entrainment, particularly in the basic acoustic/prosodic parameters
%such as speaking rate, pitch range and mean, intensity, and duration
%of pauses, SDS systems should be able to adapt their synthesized
%output to more closely match the user's characteristics.  And if we
%find evidence of discourse-level entrainment in our data, these
%findings should also be possible to incorporate into spoken dialogue
%systems.  If a system recognizes its users' backchannel type and
%latency of use, it should be able to adopt a similar strategy itself.
%When a system detects a user's propensity for barging in or not (or
%general amount of latency in taking the turn), it may be able to adopt
%similar behavior.  If a system can recognize the cues by which the
%user gives up the turn, whether lexical or acoustic/prosodic, it
%should be able to adopt these as well.
%
However, to determine whether the entrainment we find in natural
dialogue {\em should} be incorporated into SDSs, we must first
determine that this addition will indeed improve the quality of SDS
interactions.  If so, we must then test the feasibility both of
recognizing user behaviors for systems to entrain on and of producing
system responses that are so entrained, given the state of current
speech analysis, recognition and synthesis technology.

%Developers of dialogues systems are interested in endowing their
%systems with more human like charectaristics, like the use of
%laughter and other similar non-speech noises.  In the Card Games
%corpus, laugter has been annotated and we will be able to use this
%corpus in order to test if it is a feature that induces entrainment.
%
%The use of cue phrases and the general production of topic shifts are
%another area where entrainment can happen. Adopting some of the useage
%patterns from the conversational partner could be helpful for speakers
%to achieve better communication/fewer misunderstandings. Moreover,
%this type of entrainment would be probably the easiest to integrate in
%a SDS, so in our corpus study we will focus on identifying the
%concrete mechanisms of entrainment on these aspects of conversation.
%
%
%%turn-taking behaviors
%%
%%general production of topic shifts
%%
%%disfluences, hedges
%%
%cue phrases, backchannels
%
%production of non-speech human noise (laughter, whatever)

%what else do people do that might be entrained?  semantic entrainment???
%
%
%1. Full brevity algorithm: almost selects the most distinguishing
%property available 
%
%2. Incremental algorithm: preference ordering over the properties 
%-- each entity is characterized by a list of attribute value pairs
%   (what would these be in the travel domain?)






\section{Coordination Plan} %max 3pp
\label{sec:coord}


%%roles of senior personnel
PI Julia Hirschberg, with primary expertise in speech science and
technology, will lead the project at Columbia University.  Co-PI Ani
Nenkova, with primary expertise in text-based generation, will lead
the University of Pennsylvania part of the collaboration.  This
cross-university collaboration between researchers with expertise in
text and speech processing will also contribute to bringing the two
communities closer together, by 
%%%AN providing 
engaging students at both universities
who will share research experience from both perspectives.

PI Hirschberg will have primary responsibility for supervising the
analysis and modeling of spoken phenomena, including the study of
acoustic/prosodic entrainment, and the turn-taking and
backchanneling aspects of discourse-level entrainment.  Co-PI Nenkova
will bear major responsibility for supervising the analysis of lexical
entrainment and the creation of algorithms for generating referring
expressions based upon this, and for the studies of hedging behavior
and cue phrase entrainment.  However, to maximize the educational
potential of this collaboration, students from both Columbia and Penn
will be encouraged to work together to share software, data, and
expertise across sites.   

%%how project will be managed

%%%coordination mechanisms across institutions 
The proximity between Columbia and Penn will greatly facilitate
collaboration and coordination between the two sites at minimal cost.
We will exchange monthly one-day visits to discuss recent results and
plan the necessary follow-up steps.  This can be done very cheaply
using the Chinatown Bus service which connects Chinatown in New York
City with its counterpart in Philadelphia, at a cost of \$10 per
person one-way.  For about \$500 per year we will be able to integrate
students more thoroughly into the project with face-to-face meetings,
which really do foster collaboration better than conference calls,
especially when project members will not have worked together
previously.  These meetings will be particularly useful during the
evaluation stages of the project and when we are preparing papers
based upon joint results.  During weeks when we do not have such
physical meetings, we will keep in touch via hour-long teleconferences
at a set day and time to keep up-to-date on progress at both
institutions.  The PIs and students will also meet at conferences each
year.  The budget includes appropriate amounts for bus transportation,
conference attendance (Domestic Travel) and for phone calls
(Communication).
 
In addition, we will maintain a password protected project website
which will be constantly updated, tracking most recent results, to-do
items and general project progress.  Columbia has followed this
approach in all of its research projects and found it invaluable for
managing cross-site collaborations, to keep participants informed
about stages of progress in annotation and analysis and to give
everyone access to the data.
 
The following paragraphs outline how we will allocate our time for the
research described in previous sections.

\subsection*{Year 1}
The first year will be devoted to an intensive corpus analysis of the
Games Corpus as described in Section \ref{sec:plan}. Columbia will
focus on the analysis of acoustic and prosodic features while Penn
will work on lexical entrainment. The main goal will be to identify
the features for which there is evidence that entrainment occurs, as
well as which features are correlated and what the temporal scope of
entrainment is for these features.
%Another important question that will be addressed is the
%scope of entrainment: is it the entire conversation (likely for
%lexical features) or more local, depending on only one or several
%preceding turns.  
Penn will also augment the coreference
annotations on the Switchboard corpus to allow cross-corpus
validation of the findings on the Games Corpus. Penn will also 
produce an improved prototype algorithm for
reference generation. 
We will report our findings if possible at both speech
(INTERSPEECH) and NLP conferences (International Conference on
Natural Language Generation, ACL), as well as HLT.

\subsection*{Year 2}
In the second year we will perform the perception experiments
described in Section \ref{sec:eval}.
% These
%will allow us to identify which entrainment dimensions lead to
%measurable improvement in the perceived quality/efficacy of
%interaction. 
Subjects will be recruited and experiments performed at both
institutions.  We will continue our corpus-based analyses in parallel.
In addition, we will test and refine our entrainment models on the
Switchboard Corpus.  We will report on our findings at the same
conferences noted above.


\subsection*{Year 3}
In the last year of the project we will analyze the results of our
perception studies and focus on integrating our findings into a
generation component for the \Emphasis{Let's Go} Spoken Dialogue
System, as described in Section \ref{sec:eval}.  
%Two
%types of outcomes will be sought. First, we will identify which
%entrainment models can be incorporated in the dialogue system and
%operate in real time, and for which this is not possible for the
%current state of speech technology. Second, for the currently feasible
%aspects, we will test the overall difference these make in the spoken
%dialogue system.  
We will conduct the experiments also described in Section
\ref{sec:eval} comparing the entrained versions of the generation
component with the non-entrained and 'anti-entrained' versions for
naturalness, efficacy, and efficiency.  During this year we will also
prepare our corpora for distribution (over the web if possible) to the
research community.  Again, we will report on our findings at the
conferences mentioned for Year 1.



%%% Switchboard and Meetings corpora mentioned at the end. Could be augmented if there is space
\section{The Games Corpus}
\label{sec:corpus}

The Columbia Games Corpus is a collection of 12 spontaneous
task-oriented dyadic conversations elicited from native speakers of
Standard American English (SAE) for a study of the intonational
realization of \TechTerm{given} (old, previously-mentioned)
vs.~\TechTerm{new} information \cite{Prince92}.  Thirteen subjects
(six female, seven male) participated in the study in October 2004.
Eleven of the subjects participated in two sessions on different days,
each time with a different partner.

\subsection{Design of the Corpus}

%%Subjects were paid to play a series of computer games requiring
%%collaboration between partners to achieve a joint goal.  
Subjects were paid to play a series of computer games requiring verbal
communication between partners to achieve a joint goal of identifying
and moving images on the screen.  Participants sat facing each other
in a soundproof booth with a curtain hanging between them, so that all
communication would be by voice.  Each subject was recorded on a
separate channel to a DAT recorder using a Crown head-mounted
close-talking microphone.  All games were played on separate laptops
whose screens were not visible to the other player.  All keystrokes
were captured and have been synchronized with the speech recordings
and with the items appearing on the screen at the time.

The games involved tasks of increasing complexity in terms of the
coordination necessary between the partners.  For each game, a
different set of objects appeared on each player's screen; successful
completion of the game required players to describe and discuss the
objects; they received points for each successfully completed subtask.
In some games (\TechTerm{Cards}), players saw cards with one to three
objects on them; the objects were chosen so that at least one possible
description was largely sonorant (e.g. \LingMat{loom, M\&M, mailman}),
for ease of subsequent intonational analysis.  Objects were of two
sizes (small or large) and various colors.  In other games
(\TechTerm{Objects}), only the objects themselves appeared on the
screen.  In each session, subjects were asked to play three versions
of two different Cards Games and three versions of an Objects Game;
these are described below.  The order in which objects appeared on the
screen was manipulated so that the same object reappeared at different
intervals during the game, and the number of \TechTerm{given} objects
on the screen at any time was varied systematically.
%Subjects were told that this was a ``communications
%experiment''
%%?? game-playing?  what?
%%AG: the script we read at the beginning of each session started like this:
%%    "Today we would like you to participate in a communications experiment,
%%         which will involve playing an electronic game with a partner. We will be
%%         recording your comments to one another while you play the game. You will
%%         receive online and oral instructions on how to play the game and then will
%%         be given a chance to practice before the actual experiment begins. Feel
%%         free to ask us questions at any time."
%and were given no instructions other than the rules of the game.  
Subjects were told that their goal was to accumulate as many
points over the entire session as possible, since they would be paid
additional money for each point they earned.  Subjects spoke with one
another quite spontaneously throughout the tasks.  Sample screens for
the various game types are shown in Figure \ref{figure:1-2.cards}

\begin{figure}[ht]
  \centering
  \includegraphics[width=6.4in]{screenshots-3.eps}
  \vspace{-2mm} \caption{Sample Screens from the Columbia Games Corpus.}
  \label{figure:1-2.cards}
\end{figure}

In the first type of Cards game, each player's screen displayed a pile
of 9 to 12 cards.  Player 1 was asked to describe the top card on her
pile, while Player 2 was asked to search through \Emphasis{his} pile
to find the same card, clicking a button to indicate success.  This
process was repeated until all cards in Player 1's deck were matched.
%%JH: did they get points for this??
%%AG: nope
In the second type of Cards game, each player saw a board of 15 cards
on the screen, all initially face down.  As the game began, the first
card on one player's (the \TechTerm{Describer}'s) board was
automatically turned face up.  The Describer was told to describe this
card to the other player (the \TechTerm{Searcher}), who was to find a
similar card from the cards on his board.  If the Searcher could find
a card depicting one or more of the objects described by the
Describer, the players could decide whether to declare a match and
thus receive points proportional to the numbers of objects matched on
the cards.  At most three cards were visible to each player at any
time, with earlier cards being automatically turned face down.
Players switched roles after each card was described and the process
continued until all cards had been described.  Subjects were given
additional opportunities to earn points, based on other
characteristics of the matched cards, to make the game more
interesting and to encourage discussion.

In the final game type (Objects games), each player's laptop displayed
a gameboard with 5-7 objects.  Both players saw the same set of
objects at the same position on the screen, except for one (the
\TechTerm{target}).  For the \TechTerm{Describer}, the target appeared
in a random location among other objects on the screen.  For the
\TechTerm{Follower}, the target appeared at the bottom of the screen.
The Describer was instructed to describe the position of the target on
her screen so that the Follower could move his representation to the
same location on his own.  After players negotiated their best
location match, they were awarded 1-100 points based on how well the
Follower's target location matched the Describer's.  The game
proceeded through 14 tasks, with Describer and Follower alternating
roles with each new task.

\subsection{Annotation}

Twelve sessions, totalling 9h 8m of dialogue were
recorded, of which 1h 14m correspond to the first type of Cards game,
3h 35m to the second, and 4h 19m to the Objects game.  On average, the
first Cards game took 2m 3s, the second, 5m 58s, and the Objects game,
7m 12s, averaging 45m 39s of dialogue per session.
%
%Additionally, before the actual games subjects played one short version 
%of each game to become familiar with the environment. 
%The curtain was removed during these `pre-games', so there could be
%visual communication between the players, and they were allowed to ask
%questions to the comfederate. The total duration of the pre-games is
%1h 50m.
%
Each session was downsampled to 16K.  All files in the corpus were
orthographically transcribed and words were aligned by hand by trained
annotators in a ToBI \cite{ToBI92,ToBI94,Jun05} orthographic tier
using Praat \cite{Praat} to manipulate waveforms.  Certain non-word
vocalizations, including laughs, coughs and breaths, were marked in a
ToBI miscellaneous tier, together with speech disfluencies and self
repairs.  The corpus contains 2241 unique words, with 73,844 words in
total.  It is currently being intonationally transcribed using the
ToBI conventions; 6h 2m of speakers from the Objects games
%%AG: these are single-speaker hours, while the previous values are
%%    dialogue. it may be confusing.
has been annotated to date.  Pitch, energy and duration information
has been extracted for the entire corpus automatically, using Praat.

%%In addition to the orthographic and ToBI labeling, t
The corpus is also being labeled for additional phenomena, including
\TechTerm{cue} (discourse) and \TechTerm{non-cue} (literal) use of
some \TechTerm{discourse markers}, \TechTerm{turn-taking} behavior,
and the form and function of all questions.  All lexical items
potentially indicating agreement, (e.g.  \LingMat{alright, gotcha,
  huh, mm-hm, okay, right, uh-huh, yeah, yep, yes, yup}) have been
labeled by three annotators, who separately determined whether each
item was used to indicate acknowledgment/agreement, to mark the
beginning or ending of a discourse segment, to indicate both
acknowledgement/agreement and discourse segmentation, to backchannel,
to stall in order to keep the floor, to check the interlocutor's
state, to signal the completion of a task, or as a literal modifier.
%Inter-labeler reliability for this task was measured by Cohen's
%$\kappa$ at 0.74 for the Cards games.
%this was just for the 3 labels? or? 
%%AG: This was for the 3 NU labelers, who labeled the cards games only.
%%    The kappa for the CU labelers (objects games only)
%%    was 0.51, but we have discarded those annotations, and the NU 
%%    guys (melissa, ronen, chris) are relabeling that part.
%%    As of Oct 20, all 12 objects files have been labeled by melissa
%%    10 by ronen, and 4 by chris.
%%    For those 4 files, the kappa between the three labelers is 0.62.
%%    I added that value here as an estimate of the actual Kappa.
%%    The Kappa between melissa and ronen for the 10 files that ronen
%%    has labeled is 0.60, so the estimate seems good.
%and 0.62 for the Objects Game part.
%The most common of these discourse markers in the corpus are {\it
%okay} (2439 occurrences), {\it right} (1275), {\it yeah} (974) and
%{\it mm-hm} (503).

Turn exchanges in the Objects games have been manually classified into
seven categories, following \cite{Beattie82}.  These were: \TechTerm{smooth
  switches, overlaps, butting-ins, interruptions} with and without
  overlap, and \TechTerm{backchannels} with and without overlap.  There are
approximately 2100 speaker turns in the Objects games and 1700 in the
Cards games.  
%One labeler classified all turns in the Objects Games,
%after initial experiments to identify inter-labeler reliability on a
%smaller portion of the corpus.  Again, we developed a short labeling
%manual for this task.
%All questions in the Objects games are currently being labeled for their form
%and function.
%Since questions often signaled the end of a turn we also devised a
%manual for labeling questions in the corpus and for classifying their
%form and function.  Two linguist collaborators labeled all questions
%in the Objects Games, and three trained annotators are currently
%classifying their form and function according to our manual.
%%maybe we should put all manuals on the games website and also open
%%the website to the world for a while at least. 
%%[AG: done - i created a special version for this only.
%%     it'd be confusing to show the actual page --there are so 
%%     many things there!]
All manuals for these annotations
%created for the labeling tasks mentioned above 
are available at 
%\\
\verb"http://www.cs.columbia.edu/~agus/games-project/".

\subsubsection{The Switchboard Corpus}
While we will develop our models of entrainment from our analyses of
the Games Corpus, we will test our models on another corpus, to see if
our findings generalize beyond task-based dialogues.  For this purpose
we will examine the Switchboard corpus \cite{godfrey92}, a collection
of recordings of telephone conversations on a set of pre-assigned
topics, such as favorite types of music or the new roles of women in
society.  The corpus as a whole has been orthographically transcribed,
and a subset of 12 conversations (14,555 words) has been richly
annotated with linguistic information such as pitch prominence,
%(pitch, emphatic, nuclear accent or no prominence), 
dialog acts (e.g. hedge, question, agreement, statement), the
given/new status of referents, and \TechTerm{animacy} \cite{calhoun05,nissim03}.

%\subsubsection{Meetings}
%Both the Cards Game corpus and the Switchboard corpus involve only two
%participants. In any situations there are more than two speakers
%involved: one such situation is meetings (e.g. faculty meetings, research
%group meetings). In those situations the presence of multiple
%conversation participants could lead to new or modified forms of
%entrainment. We will use the available meetings corpora \cite{Murray06b,Galley06} as a final
%validation of our findings. 
%
%
%%% -*- LaTeX -*-
%%% $Id: description.tex,v 1.36 2005/04/29 23:12:08 martin Exp $
\documentclass[11pt]{report}

%%AG: I added these
\usepackage{amsmath,epsfig}

%%\pdfoutput=0  %% asks pdflatex to generate DVI output instead of PDF

\ifx\pdfoutput\undefined
  \relax
\else
  \pdfoptionpdfminorversion=2
  \pdfcompresslevel=9
\fi

\usepackage{type1cm}
\usepackage[activate]{pdfcprot}

%% modified ot1ppl.fd, scaled Palatino by 5%
\DeclareFontFamily{OT1}{ppl}{}
\DeclareFontShape{OT1}{ppl}{m}{n} {<-> [1.05] pplr7t}{}
\DeclareFontShape{OT1}{ppl}{m}{sc}{<-> [1.05] pplrc7t}{}
\DeclareFontShape{OT1}{ppl}{m}{sl}{<-> [1.05] pplro7t}{}
\DeclareFontShape{OT1}{ppl}{m}{it}{<-> [1.05] pplri7t}{}

\DeclareFontShape{OT1}{ppl}{b}{n} {<-> [1.05] pplb7t}{}
\DeclareFontShape{OT1}{ppl}{b}{sc}{<-> [1.05] pplbc7t}{}
\DeclareFontShape{OT1}{ppl}{b}{sl}{<-> [1.05] pplbo7t}{}
\DeclareFontShape{OT1}{ppl}{b}{it}{<-> [1.05] pplbi7t}{}

\DeclareFontShape{OT1}{ppl}{bx}{n} {<->ssub * ppl/b/n}{}
\DeclareFontShape{OT1}{ppl}{bx}{sc}{<->ssub * ppl/b/sc}{}
\DeclareFontShape{OT1}{ppl}{bx}{sl}{<->ssub * ppl/b/sl}{}
\DeclareFontShape{OT1}{ppl}{bx}{it}{<->ssub * ppl/b/it}{}

\usepackage{mathpazo}

\usepackage{geometry}
\geometry{margin=2.5 true cm, nohead, footskip=15bp, includefoot}

%\renewcommand{\baselinestretch}{1.05}
\parindent=1em
\hyphenpenalty=0
\widowpenalty=0
\clubpenalty=0

\usepackage{graphicx}
\graphicspath{{./}}

\usepackage[usenames,dvipsnames]{color}
\definecolor{Black}{rgb}{0,0,0}
%%\ifx\useColor\relax
\ifx\useColor\relax
  \definecolor{Linkcolor}{rgb}{0.7,0.0,0.0}
  \definecolor{Citecolor}{rgb}{0.0,0.5,0.0}
  \definecolor{Filecolor}{rgb}{0.0,0.0,0.5}
\else
  \definecolor{Linkcolor}{rgb}{0,0,0}
  \definecolor{Citecolor}{rgb}{0,0,0}
  \definecolor{Filecolor}{rgb}{0,0,0}
\fi

\usepackage[numbers]{natbib}
\bibliographystyle{abbrvnat}
%%\newcommand{\citeg}[1]{\citeauthor{#1}'s \citeyearpar{#1}}
\renewcommand{\bibname}{References Cited}

\usepackage[pdfpagelabels]{hyperref}
\hypersetup{
  breaklinks=true,
  plainpages=false,
  colorlinks=true,
  anchorcolor=Black,
  linkcolor=Linkcolor,
  menucolor=Linkcolor,
  pagecolor=Linkcolor,
  citecolor=Citecolor,
  filecolor=Filecolor,
  urlcolor=Filecolor,
  bookmarksopen=true,
  bookmarksnumbered=true,
  pdftitle={RI: Collaborative Research: Speaking More Like You:
    Lexical, Acoustic/Prosodic, and Discourse Entrainment for
                Spoken Dialogue Systems},
  pdfauthor={Hirschberg et al.},
%
%  pdfsubject={IIS - Collaborative Systems; IIS - Advanced Learning Technologies},
%  pdfkeywords={prosody, language training, spoken language systems},
  %%pdfpagemode=None,
  pdfstartview=FitH
}

\usepackage{hypcap}

\usepackage{rcs}
\setlength{\fboxsep}{8pt}
\newcommand{\CVSinfo}{\smallskip\framebox[\linewidth]{File:
    \RCSRCSfile{}, Revision: \RCSRevision{}, Date: \RCSDate{}
    \RCSTime\,\textsc{utc}}\bigskip}

\usepackage{comment}
\specialcomment{meta}{\medskip\hrule\medskip}{\hspace*{1pt}\hrule\medskip}
\specialcomment{CVS}{}{\CVSinfo}
\excludecomment{meta}
\excludecomment{CVS}

\usepackage{paralist}

\usepackage{ragged2e}

%% Page and section numbering

\renewcommand{\thechapter}{\Alph{chapter}}
\renewcommand{\thepage}{\Alph{chapter}--\arabic{page}}

\setcounter{secnumdepth}{4}  %% number headings down to subsubsections
\setcounter{tocdepth}{4}     %% and put them in the toc

\makeatletter
\renewcommand{\chapter}{%
  \if@openright\cleardoublepage\else\clearpage\fi%
  \global\@topnum\z@%
  \@afterindentfalse%
  \phantomsection%
  \setcounter{page}{1}%
  \refstepcounter{chapter}%
  %%\addtocontents{lof}{\protect\addvspace{18\p@}}%
  %%\addtocontents{lot}{\protect\addvspace{18\p@}}%
  \secdef\MYchapter\MYsChapter}

\newcommand{\MYchapter}[2][?]{%
  \addcontentsline{toc}{chapter}{\protect\numberline{\thechapter}#1}%
  \vspace*{0pt}%
  {\centering\sffamily\bfseries\huge\thechapter\quad #2\par}%
  \vspace*{4ex}%
  }
\newcommand{\MYsChapter}[1]{%
  \addcontentsline{toc}{chapter}{\protect\numberline{\thechapter}#1}%
  \vspace*{0pt}%
  {\centering\sffamily\bfseries\huge\thechapter\quad #1\par}%
  \vspace*{4ex}%
  }
\makeatother

\usepackage[sf,bf,small,compact]{titlesec}

% \titleformat{\section}[wrap]
% {\normalfont\sffamily\bfseries\fontsize{11pt}{14pt}\selectfont\filright}
% {\thesection}{.5em}{}
% \titlespacing{\section}{12pc}{1.5ex plus .1ex minus .2ex}{1pc}

\titleformat{\subsection}[runin]
{\normalfont\sffamily\bfseries}
{\thesubsection}{.5em}{}

%% local macros

\newcommand{\ChapRef}[1]{\hyperref[{chap:#1}]{Chapter~\ref*{chap:#1}}}
\newcommand{\EqRef}[1]{\hyperref[{eq:#1}]{(\ref*{eq:#1})}}
\newcommand{\FigRef}[1]{\hyperref[{fig:#1}]{Figure~\ref*{fig:#1}}}
\newcommand{\SecRef}[1]{\hyperref[{sec:#1}]{\S\ref*{sec:#1}}}
\newcommand{\PageLabel}[2]{\hypertarget{p:#1}{\label{p:#1}#2}}
\newcommand{\PageRef}[1]{\hyperlink{p:#1}{page~\pageref*{p:#1}}}

%\let\Acronym=\textsc
%\let\Concept=\emph

\newcommand{\YN}{\emph{yes\slash no}}

\newcommand{\FIXME}[1]{\textbf{\textcolor{red}{#1}}}

\clubpenalty=9900
\widowpenalty=9900

%%AG: redefine the 'quotation' environment (used in pilots.tex)
\renewenvironment{quotation}
               {\list{}{
                                        \topsep 1mm
                                        \parsep 0mm 
                                        \itemindent 0mm
                                        \leftmargin \parindent
                                        \rightmargin \parindent
                                }
%                                \fontfamily{phv}\selectfont
				\it
                \item\relax}
               {\endlist}

%%AG: format of technical term, emphasis 
%%    and linguistic materials
\newcommand{\Emphasis}[1]{{\bf \em #1}}
\newcommand{\TechTerm}[1]{{\sc #1}}
\newcommand{\LingMat}[1]{{\it #1}}

\begin{document}

\setcounter{chapter}{4}


\input{intro} %(*JH,all) (1p or less)
%%JH: mention collab with kth somehow?
\input{previous} %prev work on entrainment and IVR response generation (1p?)
%\input{goals} % (.5p) do we really need this or is intro enough?
\input{corpus} %description of GAMES corpus and annotations (1-2pp)
\input{pilots} %stuff on referring expressions, cue phrase use and hedges
\input{analysis} %proposed analyses of corpus wrt acoustic/prosodic
                 %(features such as contour use, choice of accent type,
                 %realization of given information), turn-taking
                 %behaviors, and lexical usage (e.g. referring expressions)
%\input{lab_studies} %should we mention proposed collaboration with tillburgh?
%\input{sds} %test out findings in a system
\input{evaluation} % put this into analysis and sds sections
%\input{education} % course material development?  tutoring systems???
\input{prevnsf} % results of previous nsf funding most related:
                % games/bdc, emotion,tutoring, translating (autobi)

%\input{conclusion}  %
\input{impact} %conclusions and impact on the world (1p)
\clearpage
\input{coord}

\clearpage
\RaggedRight
\hyphenpenalty=9999

%\bibliography{refs,refs-previous,/home/julia/bib/refs}
\bibliography{refs,refs-previous,refs-julia}
%%AG: /home/julia/bib/refs.bib has a few errors. I copied it to refs-julia.bib
%%    and corrected them.

\end{document}
\section{Curriculum Development Activities}
\ref{sec:curric}
                     EDUCATION/CURRICULAR DEVELOPMENT
        
        motivation: most nlp faculty do not have expertise in spoken
        language processing and thus cannot train their students in
        this area.  a vicious circle.  we propose to develop and test
        a component of a spoken language processing course focussed
        upon spoken dialogue, based upon use of the games corpus and
        its annotations, and associated technologies.

                for homeworks, students will analyze the data using
                annotations we have provided (e.g. what is the
                correlation between turn length and X?)

                students will build a spoken dialogue system (tts,
                dialogue management, asr) as group projects
                
                etc

                schedule:  year 1 develop curric (check it out in lsa
                course?)
                year 2: send to penn and harvard, evaluate
                year 3: iterate
                after: make available on web to all

                evaluation by survey, compared to other components
                
                letters:  dan j, etc.
\subsection{Model validation: reproducing human decisions}

Our initial perception studies described in the previous section will
lead to the identification of lexical and acoustic parameters that can
be entrained on and that lead to perceived improvement in the dialogue
naturalness. The next question that needs to be addressed is can we
build an automatic model that {\em predicts} human entrainment
behavior. For example, if the perception experiments demonstrate that
rate of speech and referring expression extrainment do lead to more
natural conversations, than the next question is how to compute
appropriate speaking rate and referring expresion. The off-line
experiments outlined in this section will test the fit between our
models and actual human productions.

For the purpose, we will use a held out portion of our Cards Game and
Switchboard corpus. For each utterance, we will compute the
probability of it's realization using a general model and a context
sensitive (entrainment) model. For example for speaking rate, the general model
would predict a speaking rate which is the average speaking rate of
the speaker in the entire conversation ($R_a$). The entrainment model would
predict a different speaking rate ($R_e$), conditional on the dialogue
preceeding the current utterance, which would be a function of the
speaker's average speaking rate and the speaking rate of their
conversational partner. Then we can compare which of $R_a$
and $R_e$ is closer to the speaking rate actually produced by the
speaker. 

In a similar way, models for lexical entrainment can be tested through
generating a reference either using a standard referring expression
generation algorithm  or using our augmented models that incorporate
preference for distinguishing properties and lexical items based on
the part of the dialogue preceeding the taget utterance. The
automatically produced expressions under both conditions can be
compared with the actual production in the dialogue and success will
be measured by the number of of referring expressions used by the
person that are faithfully reporduced by the respective
algorithm. This approach has already been been adopted for evaluation
of first references in written (newspaper articles) text \cite{SidhCops04}. 

Such off-line processing evaluation paradigm will allow us to compare
different possible models of entrainment and choose the ones that best
fit the human data. 

The next step in our evaluation would be to integrate the best
performing models into a fully functioning spoken dialogue
system. [And that section comes next here]\section{Testing and Evaluation} 
\label{sec:eval}

Because of the complexity of the entrainment phenomenon and the
diversity of types of entrainment we are examining, we propose a suite
of evaluation scenarios to help us classify the kinds of entrainment
that can be detected in human interaction and their respective
significance. Our corpus-based analyses described in Section
\ref{sec:plan} will provide a set of parameters for which we have
evidence of entrainment from human-human data.  Three very important
questions arise next:
\begin{enumerate}
\item {\bf What makes a difference?} Of the entrainment types that we
  have identified in the corpus-based study, which ones actually
  contribute to improving dialogue naturalness and effectiveness? It
  is possible that humans entrain to other humans in ways that may not
  influence the efficacy or perceived quality of communication in
  measurable ways.  We will use perception experiments to identify
  which types of entrainment lead to a perceived difference in dialogue
`success' in these areas.
\item {\bf What is the best model?} We need a computational model for
  the entrainment types identified as important for dialogue
  quality. In order to find the best models and to compare their
  effectiveness with simpler baselines, we will test how well they fit
  held-out human-human dialogues.  How do our models predict real
  human productions?
\item {\bf Can the best models be integrated in current spoken
    dialogue systems?} Given the best models identified in the
  reproduction of human dialogue experiments, we next need to assess
  the feasibility of integrating these models into real SDSs.  Some of
  the models may be too computationally intensive to be incorporated
  into real SDS.  Others might demand access to information about
  human partners that is not reliably capturable using current speech
  analysis tools, or the ability to modify system outputs in ways that
  cannot be achieved effectively given current language generation
  technologies.  This third round of experiments will establish how
  much of our findings \Emphasis{can} be incorporated into SDSs, given
  the current status of speech technology, and which models would pose
  challenges that would require future technological development.
\end{enumerate}

\subsection{Perception Study: Prerecorded Interaction}
\label{percep1}

In order to identify which types of entrainment correlate with
improvement in dialogue quality, we will run perception studies in
which subjects are asked to rate the naturalness of a set of simulated
conversations between a human speaker and a dialogue agent.  The
utterances corresponding to the human speaker will be natural
productions extracted from the Games Corpus, while those corresponding
to the agent will be generated with a text-to-speech (TTS) system
under several conditions.  Note that in all experiments we will use
one of the Festival \cite{Festival}, Cepstral \cite{Cepstral}, or IBM
\cite{IBM-TTS} TTS systems, all of which are available at Columbia,
for system output.

In the agent's utterances, we will systematically vary acoustic,
prosodic and lexical features that have been identified in our
corpus study as possible parameters for entrainment. The agent's
response will be generated under three conditions, simulating
either (1) an entrainment behavior with respect to the
feature being studied, (2) the lack of entrainment (neutral with
respect to what an entrainment model would predict), or (3) an
`anti-entrainment' variant of the feature, in which the feature
changes in a direction opposite to the one predicted by entrainment.
%%AG: this is not clear... i can't find the correct phrasing.
%%AN I tried to clarify, check to see if it is better
Subjects will be asked to rate the naturalness of each conversation
condition using a 1-5 Likert scale. These ratings will provide
evidence of which types of entrainment improve the naturalness of
conversations.

%%AG: trans-cultural analysis? subjects: native speakers of mandarin

\subsection{Perception Study: Wizard-of-Oz Experiments}
\label{percep2}
The purpose of this evaluation will be the same as the experiment
described in Section \ref{percep1}, but the experimental design will
simulate a setting closer to real SDSs.  In this experiment,
%JH: do we want to say this is a new set of subjects here?
subjects will perform a series of collaborative tasks using a
computer-based dialogue agent, controlled by one of the experimenters.

In an initial calibration phase, each subject will be asked to
describe a set of objects to the agent. The subject's speech will be
recorded and a set of lexical, acoustic/prosodic and discourse-level
parameters will be measured from the speech.  The subject will be
given a set of distractor tasks to perform while the measurements are
being made.  These measurements will be used to modify parameters of
the TTS system in one of three ways, as in the perception study above:
to simulate entrainment, lack of entrainment, or anti-entrainment.
The subject will then be asked to perform a set of short,
collaborative tasks with the dialogue agent (e.g. a modified version
of the Objects game).  Since these tasks will be specified in advance,
we can predict a small set of appropriate responses for each of them.
The Wizard can then rapidly select appropriate agent responses by
button click from a pool of responses automatically generated with the
appropriate parameter settings for the desired entrainment condition,
to avoid any unusual response latency that might be incurred by typing
in to the TTS system.  Each subject will be tested in all
three conditions during different stages of the experiment, balancing
order of condition and task across subjects.

Subject performance on the tasks will be measured in terms of time to
task completion and quality of task completion.  At the end of each
task, the subject will be asked to perform a series of subjective
ratings of system performance, testing the naturalness of the
conversation.
%JH: we should refer to or mention all those questions asked by the
%paradise people 

\subsection{Model Validation: Reproducing Human Decisions}
\label{model}

Our initial perception studies (Sections \ref{percep1} and
\ref{percep2}) will identify parameters that can be entrained to other
speakers and that lead to improvement in the efficacy, efficiency and
perceived naturalness of dialogue interaction.  The next question to
address is: Can we build an automatic model that \Emphasis{predicts}
human entrainment behavior?  For example, if our perception
experiments demonstrate that speaking rate and referring expression
entrainment \Emphasis{do} lead to more natural conversations, then the
next issue is how to compute appropriate speaking rates and referring
expressions. The off-line experiments outlined in this section will
test the fit between our models and actual human productions.

First, we will divide the Games and Switchboard corpora into training
and test sets. For each utterance in the test set, we will compute the
probability of its realization using first a general model and then an
entrainment model trained on training data for that corpus.  For
example, the general model would predict a speaking rate conditioned
only on the current speaker, such as the average speaking rate for
that speaker over the entire conversation ($R_a$). The entrainment
model might predict a different speaking rate ($R_e$), conditioned on
the conversational partner's rate as well as the current speaker's,
perhaps varying as a function of time over the conversation.  We can
then compare which of $R_a$ and $R_e$ is closer to the speaking rate
actually produced by the speaker in each test utterance.

Similarly, models for lexical entrainment can be tested by generating
a reference at time $T_1$, first using a standard GRE algorithm and
then using an entrainment-driven procedure sensitive to the prior
context.  The automatically produced expressions under each condition
can be compared to the actual productions in the dialogue. Success
here will be measured by the number of referring expressions used in
the actual dialogue that are faithfully reproduced by each algorithm.
This approach has been successfully used in evaluating first
references in text newspaper articles \cite{Sidh04}.  Such an
evaluation paradigm will allow us to compare different possible models
of entrainment and choose those that best fit the human data.


\subsection{Spoken Dialogue System}

%The final experiment is very similar to the previous one, but using a
%Spoken Dialogue System instead of a WOZ system. The SDS output will be
%manipulated in each task so as to convey the different types of
%entrainment mentioned above. Again, subjects will rate the naturalness
%of the conversation in each task.
%%AG: too short?
The next step in our evaluation will be to integrate the best
performing models identified in Section \ref{model} into a fully
functioning SDS.  The purpose of this experiment is to discover which
of our models can operate in real time with current speech
technologies.  We will perform experiments similar to those described
in Section \ref{percep2}, but this time in a simple but fully
automated SDS, without participation from a human operator. We plan to
implement this experiment using components of the CMU \Emphasis{Let's
  Go} \cite{letsgo} system, which Alan Black (see attached letter)
will make available to us for this purpose.


\subsubsection{First example: ``extraterrestrial''}

In the first example, subject s11 uses the word ``extraterrestrial'' twice
to refer to a character:

\noindent {\bf s11:} okay in the middle of the card I have an {\bf extraterrestrial} figure

\noindent {\bf s11:} okay middle of the card I have the {\bf extraterrestrial}

A few minutes later, both s10 and s11 use the same expression in the following exchange:

\noindent {\bf s10:} I've got the blue lion with the {\bf extraterrestrial} on the lower right\\
\noindent {\bf s11:} okay I have the {\bf extraterrestrial} now and then I have the eye at the bottom right corner\\
\noindent {\bf s10:} my {\bf extraterrestrial}'s gone

This is s10's and s11's first session, and none of them uses the more common 
word ``alien'' to refer to that character. 
Notably, both s10 and s11 start using the same term in their second sessions. 
However, both end up yielding to their interlocutors' choice, the shorter ``alien'',
as can be observed in the following two exchanges.

\noindent {\bf s03:} okay I have a blue lion and then the {\bf extraterrestrial} at the lower right corner\\
\noindent {\bf s11:} mm \# I'll pass \# I have the {\bf alien} with an eye in the lower right corner\\
\noindent {\bf s03:} um \# I have \# just the {\bf alien} so I guess I'll match that

\noindent {\bf s10:} yes now I've got that {\bf extraterrestrial} with the yellow lion and the money \\
(...)\\
\noindent {\bf s12:} oh now I have the blue lion in the center with our little {\bf alien} buddy in the right hand corner\\
\noindent {\bf s10:} with the {\bf alien} buddy so I'm gonna match him with the single blue lion \# okay \# I've got our {\bf alien} with the eye in the corner

And after both exchanges, there are further uses of ``alien'' by s10 and s11,
but none of ``extraterrestrial''.

% ==========

\subsubsection{Second example: ``scared M and M''}

In our second example, subject s13 describes a character as a ``scared M and M'', which is
copied by s12. Interestingly, on one occasion s12 replaces ``scared'' with ``frightened''.

\noindent {\bf s13:} {\bf the orange M and M looking kind of scared} and then a one on the bottom left and a nine on the bottom right

\noindent {\bf s13:} alright I have the exact same thing I just had it's {\bf an M and M looking scared that's orange}

\noindent {\bf s13:} yeah {\bf the scared M and M guy} yeah

\noindent {\bf s12:} framed mirror and {\bf the scared M and M} on the lower right

\noindent {\bf s12:} and it's to the right of {\bf the scared M and M} guy

\noindent {\bf s12:} yeah and the iron should be on the same line as {\bf the frightened M and M} kind of like an L

\noindent {\bf s12:} to the left of {\bf the scared M and M} to the right of the onion and above the iron

Neither s12 nor s13 carried this term on to their second session.

% ==========
\section{Conclusions and Broader Impact}
\label{sec:impact}
Speaker entrainment is an important aspect of human-human
communication.  However, our understanding of the full range of
entrainment features is still in its infancy.  
We propose a study of many varieties of speaker entrainment in spoken
dialogue for the purpose of improving the naturalness, efficiency and
efficacy of SDSs and expanding their population of use.  We have
examined evidence of entrainment in the experimental literature and in
the existing corpus-based studies to motivate the features we will
examine (Section \ref{sec:previous}).  We have conducted a set of
preliminary analyses of simple lexical and acoustic/prosodic features
in two corpora, the Columbia Games Corpus and the Switchboard Corpus,
which show some indications of entrainment to motivate our further
study (Section \ref{sec:pilots}).  We have outlined a research plan
for this further analysis (Section \ref{sec:plan}).  We have
described a plan for multiple evaluations of our results (Section
\ref{sec:eval}) to determine not only which features \Emphasis{should} be
incorporated into SDS, but also to determine \Emphasis{whether} such 
incorporation is technologically feasible.

\paragraph*{Intellectual Merit}
Most uses of entrainment in SDS to date have examined how speakers can
be entrained to a system.  We propose instead that systems should be
capable of entraining to their users.  In this respect, our findings
will provide guidance on which features systems can and should be able
to entrain upon and how this entrainment should and can be modeled. In
the course of our work, we will produce richer annotations
(orthographic, intonational, discourse-level) of several existing
corpora, the Columbia Games Corpus and a subset of the Switchboard
Corpus, which will be made available to the speech and language
community for research.  This data will increase the amount of
annotated spontaneous dialogue material available for research on SDS
significantly.  We will disseminate our results widely through the
presentation of papers at major speech and NLP conferences.

\paragraph*{Broader Impacts}
The broader impacts of our work fall into two categories: 1) Our work
will support a new approach to SDS interactions --- speaker
entrainment by the system, to make such systems more useful and
natural to a larger portion of the population; 2) We also will make
research in computer science and speech technologies more accessible
by students from diverse backgrounds and at different stages in their
education.

It has long been noted that people speak differently when they speak
with children, foreigners and the elderly, to facilitate
communication.  So, SDS should also adapt to the lexical,
acoustic/prosodic and discourse styles of such users as well as to the
styles of the general adult native-speaker population.  We also
believe that language teaching itself should benefit from our findings
about entrainment in American culture.  Behaviors such as tolerance
for overlapping speech (interruption), for example, differ
significantly between cultures \cite{Tannen86}; yet such behaviors are
rarely taught to second language learners.
%acceptable overlap in speech, when both speakers speak at the same
%time. Americans would generally consider overlap as impolite and would
%stop their turn if their conversational partner starts speaking before
%they have finished and signalled that they are giving up the floor;
%for Americans overlap is impolite. Italian or Jewish immigrants on the
%other hand have much longer stretches of overlap in their
%conversations and they are considered a sign that the conversation is
%interesting and going well. In cross-cultural situation a considerable
%amount of adjustemnt is necessary on part of either or both partners.

Student participation from women and minorities at graduate and
undergraduate levels will be actively sought at both Columbia and the
University of Pennsylvania.  In particular, new GRAs will be funded
under this grant at both institutions for this research, with women
and minority candidates specially recruited at both.  The PI and co-PI
have both been very active in the recruitment and retention of women
in computer science, serving on numerous diversity committees whose
aim is to increase the participation of women in engineering and the
sciences, advising student groups and organizing activities such as
panels, invited talks and workshops at Columbia to enhance women's
ability to succeed in the engineering sciences.  Numerous women
undergraduates and graduate students have worked on NSF-sponsored and
other research projects at Columbia, including 3 women master's
students in 2005-6 and a woman REU in Summer 2006.
%JH: add diversity committee stuff to bio
\section{Introduction}
\label{sec:intro}


When people engage in conversation, they adapt the way they speak to
the speaking style of their conversational partner.  For example,
they may alter their \TechTerm{speaking rate} to speak faster, if
the person they are speaking with speaks faster than they do.
%% AN the MM examples was very surprising at this place; I tried to
%% include more prepping, but this made it more verbose.
Or they may adopt a certain way of describing something based
upon the way their conversational partner describes it, negotiating a
common description, particularly for items that may be unfamiliar to
them.  For example one speaker might describe a picture of a weaving loom as ``\LingMat{the thing
that looks like a harpsichord},'' and the other might refer to this object
later as ``\LingMat{the harpsichord-looking thing}.''
This phenomenon is known in the literature as \TechTerm{entrainment}, 
\TechTerm{accommodation}, \TechTerm{adaptation}, or \TechTerm{alignment}.
We will refer to it below as \TechTerm{entrainment} for simplicity's sake.

While there is considerable experimental evidence for
\TechTerm{lexical entrainment} and some evidence of adaptation of
speaking rate in natural speech, much less is known about other types
of speech entrainment and little corpus-based work has been done.  We
propose a systematic examination of lexical, acoustic/prosodic and
discourse-level entrainment in two corpora of spontaneous speech.  Our
goal is to identify evidence of which \Emphasis{types} of entrainment
occur in spontaneous dialogue, to determine \Emphasis{how} they occur,
and to evaluate whether these entrainment phenomena \Emphasis{should}
and \Emphasis{can} be incorporated successfully into Spoken Dialogue
Systems (SDS) to improve their naturalness and usefulness.

In this study, we will examine how people adapt to their
conversational partners in acoustic/prosodic and discourse dimensions,
as well as in their lexical and syntactic decisions.  We will examine
speakers' lexical content and spoken realization of referring
expressions; their pitch, amplitude and timing variation; their
choice of intonational contours, pitch accents and phrasing
decisions; their turn-taking behaviors; the way they mark topic shift;
the way statements are `hedged'; and their use of laughter and audible sighs.
Our study will be based on two corpora: the Columbia Games Corpus, a
large corpus of spontaneous dialogues produced by subjects playing a
collaborative computer game, which has been annotated for a range of
acoustic, prosodic and discourse features; and a subset of the
Switchboard Corpus, which also has been annotated for some of the same
phenomena.

%%AN trying to get rid of multiple use of ``phenonena'' below
While there have been many studies of entrainment in human-human
dialogue and some studies of user entrainment to systems in SDS, we
still know little about how important this entrainment is to
human-human or to human-machine dialogue, particularly non-lexical
entrainment.  So we will test our corpus-based findings with a set of
perception and production studies designed to determine: 1) Which
types of entrainment do indeed make a difference in SDS in perceived
naturalness and in measures of dialogue efficiency and success?  2)
Can current SDS technologies support the types of entrainment our
studies find to be effective in these regards?  To date, SDS
researchers and developers have seen evidence of speaker entrainment
primarily as means to influence \Emphasis{users} to adopt particular
speaking styles and vocabulary, so that recognition systems might be
able to recognize user input more accurately.  However, little
attention has been paid to how or whether \Emphasis{systems} should
entrain to their users in SDS.  We believe that systems themselves
will be more successful conversational partners for a wider population
of potential users if, like humans, they attempt to entrain to users
communicative styles.

Below we describe previous work on entrainment (Section
\ref{sec:previous}).  We then describe the corpora we will analyze
(Section \ref{sec:corpus}) to discover evidence for different types of
entrainment and how they are realized in spontaneous dialogue.  Our
research plan is outlined in \ref{sec:plan} and our evaluation plan in
Section \ref{sec:eval}. We relate the proposed research to our previous
NSF-funded work in Section \ref{sec:prevnsf}. In Section
\ref{sec:impact} we explain the scientific contribution of our work
and discuss the implications of the study in a broader social context.
In Section \ref{sec:coord} we describe our plans for coordinating this
collaborative study between Columbia University and the University of
Pennsylvania.

%%% -*- LaTeX -*-
%%% $Id: one-page.tex,v 1.16 2005/04/30 12:56:11 julia Exp $

\begin{flushleft}
  \fontsize{10.8bp}{11pt}\sffamily\bfseries%
  RI: Collaborative Research: Speaking More Like You: Lexical,
  Acoustic/Prosodic, and Discourse Entrainment for Spoken Dialogue
  Systems
\end{flushleft}

\noindent% 

When people speak with others, they often \TechTerm{entrain} to their
conversational partner, adopting the partner's word choice or changing
their speaking rate to match the partner's more closely.  While much
is known about \TechTerm{lexical entrainment}, other aspects of
entrainment have received less attention.  Intonational contour, pitch
accent, phrasing rate, pitch range, loudness, production of laughter,
turn-taking and backchanneling behaviors, ways of signaling discourse
structure or hedging one's statements are all areas in which
entrainment may occur in dialogue, but they have been little studied
in this regard.  Even forms of entrainment that {\bf have} been
identified have not yet influenced response generation in Spoken
Dialogue Systems.  While proposals have been made that systems could
improve speech recognition accuracy by
taking advantage of users' preference for re-using words from system prompts
in their responses, no
system today entrains to its {\bf users'} speech.
%Do
%such entrainment phenomena occur generally in natural spontaneous
%speech as well as in laboratory experiments?  
%Which phenomena are most
%likely to be adapted when a speaker deals with someone whose behavior
%is different from her own, and which do not?
%to be invariant but speaker-specific, invariant but shared?  
%Does entrainment in one dimension tend to correlate with entrainment
%in others?  
%At what level of temporal granularity should we study
%speaker entrainment --- ; 
%How rapidly do speakers entrain to one another along the various dimensions?
%Are there regular processes by which speakers
%entrain in any dimension; e.g.  when people reach some agreement on
%how to refer to some discourse entity, is this process marked by any
%particular lexical or non-lexical cues?  What happens when speakers do
%not entrain?  Is the exchange less natural, fruitful, fluent?  
We will investigate lexical, acoustic/prosodic, and discourse-level
entrainment in two large corpora of spontaneous dialogues.
%btween people playing a computer game with different
%onversational partners (Columbia Games Corpus) and between speakers
%ngaged in topic-specified telephone conversations (Switchboard
%orpus), 
%have been annotated for a variety of prosodic and discourse-level features.
%entrainment in spontaneous speech. 
We will examine {\bf how} such entrainment occurs as well as {\bf which
features} are entrained. We will evaluate the results of our analyses
in perception experiments and in a  response generation
system for a Spoken Dialogue System.

%e.g.: Do both speakers entrain to one another, or does one typically
%adapt to the 'dominant' speaker?  Over what span of time does
%entrainment take place?  Does this differ for different communicative
%dimensions?  Which aspects of entrainment correlate with one another?
%We will build models of entrainment in these dimensions based upon our
%results and test these models in perception studies and in the
%response generation component of an SDS to determine the contribution
%that different types of entrainment play in dialogue.
%\vspace*{\fill}%
%\noindent%
%\textbf{\textsf{Intellectual Merit of the Research and Qualifications
%    of the Researchers}}\\
%\noindent
\vspace{1mm} {\bf Intellectual Merit: } To our knowledge, our research
represents the first corpus-based study of speaker entrainment to
%% AN I wouldn't say ``multiple'' since in the call they explicty
%% mention multi-party conversations such as meetings so it might
%% become confusing. ``Different'' or ``conversational'' is more neutral
%% multiple 
conversational partners in lexical, acoustic/prosodic, and discourse-level
features.  The entrainment models we develop and test will make it
possible for SDS to entrain to users, as opposed to users
to SDS.  Our perception studies will inform research on dialogue
systems as to which types of speaker entrainment are most important to
model and how they can be realized to promote dialogue naturalness,
efficiency, and efficacy.  Our findings will also impact research on
Natural Language Generation by identifying additional
conversational-partner dependent constraints on the choice of lexical
items and their spoken realization.

%Work in computational linguistics on generation of reference will also
%be improved through the incorporation of entrainment components in
%current generation algorithms. As the entrainment literature suggests,
%reference in dialogue is a collaborative process in which the two
%speakers track each others focus of attention, while current genration
%algorithms assume universal and static order of attributes. Most
%current algorithms are specialized for generation of first reference
%but in our work we will provide a unified framework to spuuport
%generation throughout the dialogue.
  

%e.  It is oriented toward system adaptation
%to user, an ability that is not available to current systems; in human
%to human communication, both participants adapt to make the
%conversation maximally successful and the lack of adaptation on the
%system's side undoubtedly contributes to communication problems. Our
%experiments will also show which kinds of adaptation can be
%implemented in current dialogue systems , given the current state of
%speech technologies. The success of our study will be greatly
%facilitated by the availability of very good resources for developing
%our theories and testing their applicability. We have collected a
%large in-house (Columbia) corpus of spontaneous task oriented speech,
%and will also have access to naturally occurring dialogues which have
%been richly annotated else where (Switchboard).
%
%% and the lack of which 
%  rather than user to system.  what things
% that people do are actually possible for machines to do, given the
% current state of speech technologies?  which areas do people adapt to
% others in?  which do they not adapt but share?  which are non-adapted
% but individually different?  we have good resources (corpora, etc)

\vspace{1mm} {\bf Researchers' Qualifications: } The PIs are well
qualified for this research: PI Hirschberg has over 20 years of
experience studying the acoustic/prosodic and discourse-level
phenomena which we will now examine for speaker entrainment in
dialogue. She also has conducted extensive research on Spoken Dialogue
Systems.  PI Nenkova has recently completed her dissertation on the
generation of referring expressions in summarization.  She has
conducted postdoctoral researchon prosodic variation in dialogue on
the Switchboard Corpus at Stanford. Both PIs have extensive experience
designing and conducting perception and production studies on text and
speech and evaluating their results; e.g., PI Nenkova is an inventor
of the \TechTerm{Pyramid} method of summarization evaluation.
%\vspace*{\fill}
%\noindent%
%\textbf{\textsf{Broader Impact}}\\
%\noindent%

\vspace{1mm}
{\bf Broader Impact: } While the study of entrainment is important to
our understanding of human communicative behavior, it is also of major
practical importance to the future of Spoken Dialogue Systems.
%To
%  date, research on human entrainment has primarily been seen as a
%  means to influence users' choice of vocabulary or speaking rate in
%  order to improve speech recognition accuracy: If users can be led to
%  use particular words or syntactic constructions, or to say these in
%  a faster or slower rate, automatic speech recognition engines may be
%  able to recognize their speech more accurately.  Little attention
%  has been paid to the notion that systems themselves might be more
%  successfully conversational partners if, like humans, they could
%  entrain to the communicative styles of their users.  We will
%  evaluate results of our corpus-based analyses to determine: 1) Which
%  types of entrainment make a difference in SDS in terms of increased
%  naturalness, efficiency, and efficacy?  2) Can we build automatic
%  procedures to {\bf predict} entrainment parameters?  and, 3) Can we
%  implement such predictions and realize their predicted behaviors in
%  current SDSs?
Our research will support an important new dimension in dialogue
systems, the notion that systems should adapt their behavior to users,
rather than vice versa.  Given that intelligent systems are becoming
more and more popular in daily life and activities, such a capability
is important for improving the human-machine experience for average
users as well as for users who differ significantly from the majority
population in terms of cultural or language background or age. Also,
as a by-product of our research, we will make two richly annotated
corpora available to the Natural Language Processing (NLP) and speech
communities to allow others to participate in future research on
spoken dialogue.  Our results will be disseminated through papers
presented at speech and NLP conferences.

\vspace{1mm} {\bf Broadening Participation: }The PIs are committed to
broadening the participation of under-repre\-sented groups in Computer
Science and will use this research collaboration to that end.  Women
and minority students at both institutions --- at the undergraduate,
master's and PhD levels --- will be recruited to participate in this
research. PI Hirschberg has mentored women undergraduate interns at
Bell Labs and AT\&T Labs, three of whom have received CS PhDs, and
regularly includes women MS and undergraduates in research projects,
including an REU student in 2006.  She serves on three university
committees for recruitment/retention of a more diverse faculty and
advises two women's student organizations and Barnard CS majors.  PI
Nenkova was president of WICS 2002--05 and the founder of WICSE, an
organization of women engineering students at Columbia, receiving a
departmental award for extraordinary service in 2005.

\vspace{1mm}
\textbf{Key Words:  entrainment; prosody; Spoken Dialogue Systems; Natural
  Language Generation.}
\endinput
\section{Pilot Studies}
\label{sec:pilots}

In the corpora described above (Section \ref{sec:corpus}), we have found
%some indications 
evidence of entrainment of various types.
%%Below we describe
%%some of these, as motivation for more detailed analyses of these corpora.
Below we present some examples which illustrate several types of
entrainment in the Games and Switchboard corpora and which motivate
our future research.

\subsection{Lexical Entrainment}
\label{subsec:pilots-lexent}

The Games Corpus is rich in examples of lexical entrainment, which
occurs particularly when subjects described ambiguous pictures. For
example, a picture of a loom came to be identified in one conversation as \LingMat{the
  harpsichord}; in others,
%%JH: other examples where people just didn't know what the picture was?
%%AG: I added these two
a lily eventually became \LingMat{the orchid} and a Menorah was
identifed as \LingMat{the Jewish candelabra}).  Also, when a picture
included multiple features, e.g. \LingMat{a large blue rhinoceros},
where one of more features might turn out to be salient in future
contexts, we find many negotiations of conceptual pacts.  The fact
that most speakers played with two different partners in different
sessions but viewing the same objects is especially useful for our
study, since it will allow us to compare different entrainment
opportunities for all such subjects.  For example, in the first
session for Subjects S11 and S10, S11 introduces the term
\LingMat{extraterrestrial} to refer to a picture most other users
labeled \LingMat{the alien}, and S10 accepts and re-uses this
description:
\begin{quotation}
{\rm S11:} okay in the middle of the card I have an \textbf{extraterrestrial} figure

\vspace{-1mm}
...

{\rm S11:} okay middle of the card I have the \textbf{extraterrestrial}
\end{quotation}
A few minutes later, they both confirm this conceptual
pact:
\begin{quotation}
{\rm S10:} I've got the blue lion with the \textbf{extraterrestrial} on the lower right

{\rm S11:} okay I have the \textbf{extraterrestrial} now and then I have the eye at the bottom right corner

{\rm S10:} my \textbf{extraterrestrial}'s gone
\end{quotation}
In later sessions with other partners, both S10 and S11 introduce the
same term again for the same picture.  However, in each case their new
partners reject this term for the more common description \LingMat{the
  alien}. Interestingly, both S10 and S11 immediately agree to this
new description and use it during the remainder of the session, e.g.
\begin{quotation}
{\rm S03:} okay I have a blue lion and then the \textbf{extraterrestrial} at the lower right corner

{\rm S11:} mm \# I'll pass \# I have the \textbf{alien} with an eye in the lower right corner

{\rm S03:} um \# I have \# just the \textbf{alien} so I guess I'll match that
\end{quotation}
and
\begin{quotation}
{\rm S10:} yes now I've got that \textbf{extraterrestrial} with the yellow lion and the money 

\vspace{-1mm}
...

{\rm S12:} now I have the blue lion in the center with our little \textbf{alien} buddy in the right hand corner

{\rm S10:} with the \textbf{alien} buddy so I'm gonna match him with the single blue lion \# okay \# I've got our \textbf{alien} with the eye in the corner
\end{quotation}

%The figure is never described as {\it the extraterrestrial} again.

%Similarly, Subject S13 initially describes an object as {\it an M and M
%  looking scared that's orange}, subsequently shortening this to {\it
%  the scared M and M guy} in a session with S12.  S12 accepts this
%description, although he varies the modifier in this description to
%{\it the frightened M and M} later in the task.
%
%\begin{quotation}
%{\rm S13:} \textbf{the orange M and M looking kind of scared} and then a one on the bottom left and a nine on the bottom right
%
%\vspace{1mm}
%
%{\rm S13:} alright I have the exact same thing I just had it's \textbf{an M and M looking scared that's orange}
%
%\vspace{1mm}
%
%{\rm S13:} yeah \textbf{the scared M and M guy} yeah
%
%\vspace{1mm}
%
%{\rm S12:} framed mirror and \textbf{the scared M and M} on the lower right
%
%\vspace{1mm}
%
%{\rm S12:} and it's to the right of \textbf{the scared M and M} guy
%
%\vspace{1mm}
%
%{\rm S12:} and the iron should be on the same line as \textbf{the frightened M and M} kind of like an L
%
%\vspace{1mm}
%
%{\rm S12:} to the left of \textbf{the scared M and M} to the right of the onion and above the iron
%\end{quotation}
%
%However, in this case, neither S12 nor S13 introduces this terminology
%in their second sessions, with other partners, preferring 
%the more common descriptions {\it the M and M} and 
%{\it the M and M guy}. In order to understand the differences between
%session, and the persistence of previously employed references, we
%will analyse the references (and salient properties in them) that
%preceeded our target examples to identify how certain attributes (such
%as color or state/scared) became activated in one conversation but not
%in the other. 

We also find in the Switchboard Corpus that the negotiation of a
conceptual pact appears to have interesting acoustic and prosodic
correlates; such features have, to our knowledge, not been
studied before.  For example, in the following conversation on the
changing role of women in modern society, Speaker A compares the
raising of children at home by the mother to the alternative of
organized paid child care.  Throughout the segment below, the speakers
introduce multiple `negotiable' referring expressions with
some amount of disfluency (in the example, filled pauses are
orthographically noted, `.' indicates that the previous word is
accented, and `:' indicates that the previous word has
\TechTerm{nuclear stress}, i.e., is the most prominent accent in its
intonational phrase.)
\begin{quotation}
{\rm A$_1$:} And I also. wonder. about the children: that are being. brought up: in: the uh uh day: care centers:.

{\rm A$_2$:} I guess: we'll have to see: another generation: to see. what differences. a
child. being brought: up you: know in a kind. of a: uh community: rather. than
a home:.

{\rm B$_3$:} I have. not. to be honest: had. much experience: with children: in that situation:.

{\rm B$_4$:} I guess: my experience: is is just: with what we: did and: and so they didn't: really go through the child: care route

{\rm B$_5$:} I don't: know. whether: there will be: an increased: amount. of of surrogacy: that we see: I just. don't know: 

{\rm A$_6$:} What. do you mean:?

{\rm B$_7$:} Uh: deliberate. childbirth: by surrogate. mother:

{\rm A$_8$:} Oh: yeah: 

{\rm B$_9$:} Sort of rent-a-mom. to be: you: know

{\rm B$_{10}$:} There might: be a kind. of a deliberate: uh uh. a professional: mother: a- person:

{\rm A$_{11}$:} A nanny: sort: of 

{\rm B$_{12}$:} (I can en)Vision: more women deliberately: raising children: either. in surrogacy: or:  or. as a professional nanny: nanny. as you: put it uh: 

{\rm B$_{13}$:} Maybe. we'll. see a growth. in that: where someone. makes. a career: out of say:  taking. care of five. or six. children: as opposed to day care:

{\rm B$_{14}$:} It would be. a sort of day: care but it would be more. of a family setting:

{\rm A$_{15}$:} And then you might. have more. control: over uh the the morals: that they would be. taught:  rather than: in like a classroom. or a day. care center:

{\rm A$_{16}$:} I know the day. care centers. are not. cheap: either:
\end{quotation}
In A$_{1}$, speaker A appears uncertain about how to describe the
alternative to raising children at home: his first mention of
\LingMat{day care centers} is preceded by two filled pauses. The
hesitation is also marked by increased prominence of the preposition
\LingMat{in}, which, were this production fluent, would be unlikely to
have been accented.  In the following utterance, A$_{2}$, we see that
speaker A has still not determined how best to refer to the concept of
child-raising outside the home and suggests another possible
reference, ``\LingMat{a child. being brought: up you: know in a kind
  of a: uh community}.''  Again, the whole expression is marked by
hesitations and unusual prominence assignment; a filled pause precedes
the final noun.  Speaker B at first seems no clearer on how to
describe this concept, explicitly noting that he has little experience
``\LingMat{with children in that situation}'' (B$_{3}$).  In B$_{4}$
he proposes \LingMat{child care} and in B$_{13}$ \LingMat{day care} as
alternatives.  In A$_{15}$, A accepts the latter term, and now there
is no disfluency or hesitation.

A similar reference negotiation process happens in the same
conversation with the discussion of surrogate mothers. The first
mention of the concept is in B$_{5}$, where speaker B is disfluent before
introducing the term (repeating \LingMat{of}) Speaker A explicitly
questions the meaning of this term in A$_{6}$. B's reply in B$_{7}$ is preceded
by a filled pause and his additional clarifications in B$_{9}$ and B$_{10}$, and
B$_{12}$ are marked by hesitations, hedges (\LingMat{sort of}), filled pauses
and other fillers (\LingMat{you know}, fragments and repetitions, as is
speaker A's suggestion of another term --- ``\LingMat{a nanny: sort: of}'' in
A$_{11}$.  In B$_{12}$ speaker B accepts the final suggestion, explicitly
acknowledging that he is accepting the conceptual pact.

%%% AN I am not sure how to do hedging without turning it into a future
%%% work section....

We will further study the possible relationship between incomplete
conceptual pacts and disfluencies and hesitation in dialogue. The
possibility of direct relationship between the two has significant
bearing on SDS development: if systems are not able to entrain to the
user's terminology, users might themselves become more disfluent in
subsequent utterances, making them harder for the system to
understand.  In Section \ref{sec:plan} we will describe in more detail
how we will analyze lexical entrainment in our corpora.
%A$_{15}$---a filled pause before "morals". Later on in the same
%conversation in the same conversation speaker A starts using "values",
%which seems more appropriate (I want my kids to have my values).

\subsection{Acoustic and Prosodic Entrainment}

We also noticed in the Games Corpus that some subjects do appear to
adapt their speaking style to that of their interlocutor's in terms of
their speaking rate, their choice of intonational contour and their
mean and maximum pitch.  Some initial analyses of rate and pitch,
comparing subject productions with each of the two conversational
partners they spoke with, gives preliminary indications that a given
subject's productions vary in our data to approach those of 
the partner.  Figure \ref{figure:rate} shows the mean speaking rate of
each subject in their two sessions (connected with solid lines) compared
to the mean rate of each of the subject's two partners (connected with
dashed lines).
\begin{figure}[ht]
  \centering
  \includegraphics[width=6in]{speaking-rate.eps}
  \vspace{-2mm} 
  \caption{Mean Speaking Rate (in words/sec) by Speaker Compared to Partners}
  \label{figure:rate}
\end{figure}

While mean speaking rate across an entire session is a crude measure
of potential entrainment to rate, Figure \ref{figure:rate}
\Emphasis{does} show some promising evidence of this, insofar as half
of our speakers show some evidence of rate entrainment: When subjects
converse with speakers with a faster rate, their own rate is faster
than when they converse with speakers with a slower rate.  For
example, when S01 speaks with S02, his rate is faster than when he
speaks with S09.  S02, S05, S08, S10 and S11 exhibit this pattern,
while S03's rate appears already very close to both of his partners,
and S07 and S12 are more difficult to analyze at this level.  Only two
subjects, S06 and S09 exhibit the opposite tendency, increasing their
speaking rate with a slower speaking partner.

It is more difficult to examine pitch in this simple manner because
mean pitch varies with gender.
%%, due to
%%gender differences among subjects.  
However, we see less evidence of entrainment in this feature than we
do in speaking rate,
if we examine only those subjects whose partners were only
male or only female.  Only 2 of the five with matched partners raised
and lowered their own overall pitch when speaking with partners with a
higher or lower mean.  And only one of these was among those subjects
who adapted to the partners' speaking rate.  For maximum pitch over
the session, only one speaker appeared to have been influenced by
their conversational partner.  However, interestingly, that speaker
shows evidence of entrainment to her partners in both speaking rate
and mean pitch.

Clearly, entrainment is not best measured in terms of simple measures
over long stretches of conversation.  It seems likely that acoustic
and prosodic adaptation, like lexical entrainment, is a more local or
perhaps a cumulative phenomenon.  However, these simple analyses do
suggest that some amount of measurable acoustic/prosodic entrainment
occurs in the Games corpus and that their may be interesting
individual differences not previously examined in the entrainment
literature.  Only one of our speakers adapted to her partners in all
three of the dimensions examined here, at least by our initial
metrics.  A fuller study of these and other potential entrainment
phenomena will be necessary to test whether speakers entrain to others
differently --- some in one dimension and some in another --- or
whether some forms of entrainment are more prevalent than others.
 
%\subsection{Discourse Entrainment}
%todo


\section{Previous Studies}
\label{sec:previous}

Studies of entrainment in human-human and human-machine dialogue have
largely taken place in the laboratory and have generally focused on
lexical entrainment.  However, some studies have confirmed these
findings in analyses of speech corpora and a few studies in both
laboratory and non-laboratory settings have explored the way speakers
adapt to their conversational partner's syntax, rhythm and speaking
rate, loudness, pitch range, voice quality and general style. We
briefly survey this literature, as well as some literature in
Natural Language Generation which is pertinent to our study of how lexical
entrainment might influence response generation in SDS.

\subsection{Lexical Entrainment}
\label{lex}
Influential work by a number of psychologists
\cite{Isaacs87,Garrod87,Clark89,Brennan90,Brennan96a,Brennan96c,Metzing03}
convincingly demonstrates 
%% AN we don't want to say that they demonstrate the process, rather
%% that they have shown it is there and we have to really understand
%% what the process involves?
the existence of lexical entrainment as speakers negotiate how to
refer to items in discourse. Brennan and Clark \cite{Brennan96a} term
the result of this negotiation the \TechTerm{conceptual pact} and
explore its characteristics in numerous experiments.  In particular,
they find that speakers often continue to use longer descriptions for
discourse entities in subsequent mentions than would be necessary in
order to uniquely identify the referent (e.g.  they might continue to
refer to a car as ``\LingMat{the funny yellow car},'' even when no
other cars are in view) when they have already agreed upon such a
pact.  They do find that speakers sometimes abandon such pacts and
propose potential reasons for this.  In further work, Metzing and
Brennan \cite{Metzing03} showed that deliberately violating conceptual
pacts could cause partners difficulty in locating the intended
discourse entities.  And Brennan \cite{Brennan96c} also reports on
experiments demonstrating that speakers will adopt the terminology of
partners thought to be computer systems just as readily as with
partners known to be human.  Fais and Loken-Kim \cite{Fais95},
however, found that lexical accommodation varies by interaction
modality, when subjects spoke over a phone or interacted through a
multimodal interface.  In actual human-computer interactions,
Gustafson et al.~\cite{Gustafson97} confirmed that users tended to
adapt to the terminology of system responses in general, using verbs
in system prompts more frequently, for example, than other synonyms.
Levelt and Kelter \cite{Levelt82} discovered priming effects for
syntactic constructions as well as lexical items in laboratory
experiments and similar findings are reported by Reitter et
al.~\cite{Reitter06} in corpus-based work.

Such findings have motivated attempts to improve speech recognition
accuracy in SDS by designing system prompts which will prime user
responses using words can be recognized more accurately or simply
conditioning the language model on the lexical items in system prompts
\cite{Gustafson97,Shriver00,Tomko05}.  However, there have been
remarkably few (e.g. \cite{Portele02,dialogical06}) proposals that
systems should instead entrain to their users, despite evidence that
the ability to adapt may be critical for successful dialogue; for
example, 
%task-oriented dialogues (computer repair), 
%the users calling for help were  more likely to adopt the terminology
%of the expert rather than vice versa. But
experts have been shown to adopt a user's incorrect terminology in a
computer repair exchange, in order to facilitate understanding
\cite{dialogical06}.
%in citical situations when there is a communication problem and the
%user is not able to follow the exper's direction, 
%the expert is willing to adopt the user's (incorrect) terminology in
%order to imrpove the mutual understanding 

\subsection{Acoustic and Prosodic Entrainment}

Acoustic/prosodic entrainment
%%AG: we already listed the synonyms at the beginning.
%, more often termed adaptation or accommodation, 
has to date produced fewer clear findings. The most convincing of
these have involved entrainment of speaking rate.  While some research
on rate adaptation has focused on speakers' ability to
\Emphasis{track} another speaker's rhythm when instructed to in speech
cycling and synchronous speech experiments \cite{Cummins02,Cummins05},
studies more relevant to natural dialogue have shown that speakers
\Emphasis{do} adapt to the speaking rate of others, whether the
partner is an artificial voice or another human adult or a child
\cite{Sherblom87,Guitar01,Bell03b}. Sherblom and La Riviere
\cite{Sherblom87} also found evidence for entrainment of utterance
length and vocal jitter, while Coulston \cite{Coulston02} found that
children adapted their amplitude to synthetic speech from an animated
character.  Other studies of bearing upon possible acoustic and
prosodic entrainment have documented 5-year-olds using a different
pitch range when talking to 3-year-olds \cite{Shatz73} and other
children adapting their response latencies to those of an animated
agent partner \cite{Darves02}.  However, there is no large body of
objective or experimental evidence for entrainment for most acoustic
and prosodic features in natural speech, despite the fact that
entrainment in features such as rate and amplitude could prove to be
as useful as lexical entrainment from the system perspective in SDS
systems: systems could modify their own system's rate, for example, to
encourage speakers to produce more easily recognized speech
\cite{Hirschberg04}.

\subsection{Discourse Level Entrainment}

More general studies of entrainment in multiple dimensions and in
discourse-level behavior have been primarily descriptive in nature,
including Azuma's \cite{Azuma97} account of Emperor Hirohito of Japan
adapting his `speech patterns' to those of country people in lower
social status when he visited the countryside after World War II, and
a longitudinal study by Roth \cite{Roth05} of two
\TechTerm{co-teachers} which found that one of the teaching partners
adopted catch phrases from the other and entrained to both the other
teacher's and the students' pitch and amplitude when speaking with
them.  More usefully for SDS, perhaps, Breazeal \cite{Breazeal02}
reports that in human-robot interactions people entrain to ``the tempo
of [the robot's] vocal turn-taking utterances'' so that the number of
interruptions and awkward pauses diminish over time.  When the robot
is communicating affective intent, there is evidence of entrainment by
subjects in body posture, head tilt and facial expression as well.
%
\subsection{Entrainment and NL Generation}

Research on Natural Language Generation for SDS has lagged far
behind research in recognition and understanding for such systems,
%JH: here we could put in a bit more on generation in IVR systems,
%beyond referring expressions....hmmm....there is some stuff on
%backchanneling, how to phrase clarifications and requests for
%repeats...i will dig this up....
although the generation of referring expressions (GRE) has been a
topic of significant interest for other generation and summarization
research.  The basic parameters of the GRE task are clear: how can we
describe a discourse entity so that the listener can identify it,
among other distractor entities, without including unnecessary
information that would slow down communication?  Current GRE
algorithms are deterministic in nature \cite
{dale89cooking,dale95computational, krahmer00efficient,Sidh04}, while
the entrainment literature cited above (Section \ref{lex}) suggests
that the proper form of reference to an entity is likely to be
\Emphasis{negotiated} between the conversational partners.  Moreover,
algorithms for GRE have focused on the production of initial
references to a discourse entity only
% (the linguistic expression used in a first mention of an entity in
% the discourse), 
but SDS must generate appropriate subsequent references as well. 

Recently, Viethen and Dale \cite{Dale06} have compared the productions
of several existing algorithms for GRE with actual human productions,
finding numerous mismatches between the two.  Human subjects included
more information than was necessary to uniquely disambiguate the
referent (confirming \cite{Brennan96}).  Indeed, subjects often
produced different descriptions of the same object at different times,
which current deterministic algorithms cannot do.
%If we can model not only the
%attentional state of the speaker, as well as entrainment phenomena in
%conversation, we can hope to produce natural referential behavior in
%IVR systems. 
None of the GRE algorithms could generate any of the relational
descriptions produced by people; compare one GRE algorithm's
``\LingMat{the drawer above the drawer above the drawer above the pink
  drawer}'' with a human's ``\LingMat{the orange drawer above the blue
  drawer}.''  However, some proposals have been made that GRE research
should be influenced by findings on lexical entrainment
\cite{Hirst02,HH95,DeVault05}, although these studies have focused on
the representations that would be needed to support
entrainment-influenced GRE algorithms.  Generally, the lack of
suitable corpora has so far prevented the generation community from
refining their models. And since most generation research is done on
text, few spoken language corpora have been examined with respect to
spoken GRE.  In Section \ref{sec:corpus} we describe several such
corpora which might be used for this purpose.


%GENERAL:
%
%chartrand and bargh, 1999: mimicry increases rapport
%
%bavolas, black, lemery and mullett, 1987: rapport increases mimicry
%
%goodwin, goodwin and yaeger-dror, 2002: multimodality in girls' game
%disputes, j prag 24.
%
%bock 1986: entrainment results because of lexical and syntactic
%priming
%
%garrod \& anderson 1986: entrainment results from conversational
%partners attaining a common ground
%
%schober and clark 1989: entrainment results from people attaining a
%single perspective
%
%porzel, scheffler, and malaka, 2006: entrainment annotation scheme:
%lexical, phrasal, cognitive, anaphoric, many categories of entrainment
%briefly mentioned
%
%isard, brockmann and oberlander, inlg 2006 : seem to be building 2
%systems that can 'align' with one another...
%
%hjalmarsson, 2006: utterance generation in sds.  broad overview of
%literature.
%
%*tomko, 2005:  speech shaping strategy to help speakers learn how to
%speak in a particular way for IVR system to improve interaction
%efficiency.  influenced by lexical entrainment research.  proposal but
%no content.
%
%
%CORPUS-BASED
%
%*azuma, 1997: emperor hirohito adapted his speech patterns to that of
%country folk in lower social status on visits to countryside (speech
%accommodation theory says that 'speakers will attempt to converge
%linguistically toward the speech patterns of the addressee when they
%desire social approval from the addressee, given that the perceived
%costs of so acting are proportionally lower than the rewards
%anticipated') to encourage people after ww2.  it worked.
%
%*roth, 2005: co-teachers become more like each other; (nb: concept of
%entrainment due to Christian Huygens, Dutch physicist observing 2
%pendulums that mutually influence one another).  descriptive stuff on
%word use, pitch, and volume/loudness of speech.  use of particular
%words with particular pitch contours and pausing (right; really,
%really); alignment of pitch with students and co-teacher.  pitch range
%matching observed with students. pitch contour matching. rhythm
%matching.  use some instrumentation but in a very descriptive way.
%teacher finds the 'right tone' by matching loudness and pitch to
%student's preceding turn.
%
%                        STYLE?
%
%*portele, coling, 2002: sds should adapt to the linguistic style of a
%user in analyzing input and generating output.  3 styles are defined
%but just from different corpora (verbmobile, callhome and taba.  not
%clear what the features were.  weird....
%
%                        LEXICAL/SYNTACTIC
%
%*reitter, keller and moore, 2006: syntactic priming in corpus; more
%effective in task-oriented dialogues than in spontaneous speech.
%looked for structural repetition.  hmm...very short without a lot of
%detail.
%
%*gustafson, larsson, carlson, and hellman, 1997:  effect of system
%vocabulary on lexical choices of users
%
%LAB
%
%                        TURN-TAKING/POSTURE/EXPRESSION
%*breazeal 2002
%human/robot interactions with robot Kismet that uses human-like cues:
%find that people entrain to ``the tempo of Kismet's vocal turn-taking
%utterances'' so that the number of interruptions(they mean 'overlap')
%and awkward pauses diminish over time.  when Kismet is communicating
%affective intent, there is evidence of entrainment by subjects in body
%posture, head tilt, and facial expression
%
%                        LEXICAL
%
%*shriver, black and rosenfeld, 2000: use of lexical entrainment to
%teach users 'the proper form of the command' (e.g. the movie is dadada
%instead of 'what movie did you want to see?)
%
%*brennan 1990: good review of prior lit.  entrainment reduces search
%space for objects in a domain but NLP IVR systems don't provide
%responses that help with this; importance of implicit as well as
%explicit grounding.  question: do people talk to computers as they
%talk to people wrt these conventions?  experiment:  people are told
%they are interacting with a human or with a computer. the
%conversational partner's style was varied (short answer using words
%from query; full sentence using words from query; lexical change
%response with words different from query).  then analyze whether
%people are as likely to use 3rd person pronouns (yes), first and
%second person pronouns (no), ellipsis (no), with computers as with
%humans.  also found that with both, people were  likely to entrain
%to system/human use of full sentences.  but initial strategy was full
%sentence query with humans but often key word query with computer.
%people used more direct acknowledgments in conversations with human
%partner, especially in short answer and lexical change conditions
%
%*brennan and clark 1996a: experiments showing when people refer to an
%object they are proposing a conceptualization of it, which addressees
%may or may not agree to; once they establish a 'conceptual pact', they
%use it in subsequent reference, even when a shorter expression will
%do, altho over time people may simplify the pact or abandon it for new
%conceptualizations.  this is counter to grice's maxim of quantity.
%importance of past referring behavior.  factors:  recency of last
%mention; frequency of mention; salience; provisionality (hedging,
%e.g.); partner specificity (new partner, new pact?).  results:  people
%often over-informative, lots of lexical entrainment,frequency of use
%affects entrainment, effect from provisionality (hedges) when items
%ambiguous, simple recency effect not found; partner-specific effect on
%repeated reference (continue to use over-informative expressions);
%with new addresses, speakers start wtih the old expressions but
%rapidly accommocate to new addressee.  supporting historical
%explanation for referring expression choice.
%
%?brennan 1996b:  summarizes other work
%
%*brennan, ries rubman and lee, 1996c: people interact with person
%simulating db query scheme, one with text based nl interface and one
%with asr interface.  same results for both interfaces.  people adopted
%'system' term just as often as with human partners.  effect of
%immediacy of their mention after computer's term.
%
%?brennan, psc, 2000: overview of lots of work
%
%*metzing and brennan, 2003: breaking partner-specific conceptual pacts
%results in more difficulty in getting right referent
%
%?furnas, landauer, gomez and dumain 1983, 1987: people rarely likely to
%produce same name for same computer command
%
%*clark and wilkes-gibbs, 1989; *isaacs and clark 1987, *garrod and
%anderson:  people establish local conventions for referring
%expressions
%
%?isaacs and clark 1987, milroy 1987: people's language becomes more
%simple over time as they talk with each other
%
%*levelt and kelter 1982: answers tend to contain syntactic
%constructions and lexical items which are repeated from the questions
%they follow
%
%                        TIMING/ACOUSTIC/QUALITY/PROSODIC
%
%*shatz and gelman 1973: 5 yr olds use simpler language and different
%pitch range when talking to 3 yr olds
%
%*bell, gustafson and heldner 2003: experiment showed users could adapt to
%speaking rate of animated character without being aware of the
%adaptation.  useful for getting people to speak slower, if needed to
%improve dialogue system ASR
%
%*cummins, 2002, speakers entrained by metronome to produce speech
%rhythms over short phrases; speech cycling
%
%*cummins, 2005: speech cycling and synchronous speech experiments.  in
%latter, one speaker is used to entrain another. speakers try to read a
%text together.  people can adjust their timing to one another's.
%pauses.  but weird....
%
%?watanabe, okubo, kuroda, 1997 'evaluation of the entrainment': studies
%respiration entrainment in f2f conversations.  burst-pause of speech,
%heartbeat invervals, and respiration.  physiological entrainment.
%
%?mcgarva and warner, 2003:  mutual entrainment of vocal activity
%rhythms.  experiment with female psych students paired by attitude
%similarity had 40m conversations.  speech accommodation not predicted
%by attitude similarity or attraction and did not affect ratings of
%conversation quality
%
%*sherblom and la riviere, 1987.  study of accommodation of speaking
%rate, utterance length and vocal jitter found substantial effect of
%conversational partner, interpsonal uncertainty and diffs in arousal
%level.
%
%*guitar and marchinkoski, 2001:  mother's slower speech cause 3 year
%olds to reduce their own speech rate
%
%*coulston, oviatt and darves, 2002:  children adapted to tts from
%animated characters in loudness, increasing and decreasing  (29%)
%
%*darves and oviatt, 2002: children adapted their response latencies
%18.4% to tts generated response latencies of animated partner; bi
%directional
%
%                        STYLE/PERSONALITY/REGISTER
%
%?richards and underwood 1984: polite and informal responses from a
%computer system --> users are more polite and informal
%
%?bateman and paris, 2005:  alignment via register-recognition
%
%?robertson and murachver, 2003: children accommodate to the speaking
%style of male vs female experimenters
%
%
%                        MODALITY
%
%?bell, boye, gustafson and wiren, 2000
%
%test system's referential strategy and effect on users in multimodal
%system (referring in speech or using graphics).  does this influence
%subjects use of speech vs. graphics?  limited support for 'yes'





\section{Results of Previous NSF Research}
\label{sec:prevnsf}

%%JH: this is probably going to need to be cut but won't do now

In previous NSF-sponsored research we have examined the acoustic,
prosodic and lexical characteristics of deceptive speech (NSF
IIS-0328295), finding that an automatically trained system performs
considerably better than human judges (67\% accuracy vs. a human mean
of 55\%).  We have also experimented with automatic identification of
student state (certainness, uncertainty, anger and frustration) and
detecting and classifying question-bearing turns in spoken tutoring
systems as to form and function using acoustic, prosodic and lexical
information (NSF IIS-0328295).  Here we are able to
detect whether or not a student is confident in her responses with
76\% accuracy, a 15.8\% relative improvement over a baseline always
predicting certainness \cite{Liscombe06a}.  We are also able to detect
turns that include a question with 80\% accuracy
\cite{Liscombe06b,Venditti06}.  Our goal is to emulate the behavior of
human tutors, who respond differently to students based on the
confidence they display --- when they are wrong as well as when they
are right --- and whose answers are also conditioned on the form {\em
  and} the function of student questions, which are quite frequent in
our corpus.
%% AN commented the sentence below since it is long and is not about
%% NSF funding
%These studies build upon earlier work of PI Hirschberg on SDSs in the
%automatic detection of speaker turns that are likely to be
%misrecognized, that are corrections of some system error, or that
%follow a speaker's recognition that she has been misrecognized
%\cite{Hirschberg01c,Hirschberg04,Litman06}.  We are currently
studying the relationship between pro\-sod\-ic variation across
languages (Mandarin and English) in order to develop methods of
training second-language learners of languages whose prosodic systems
may be quite different from their first language (IIS-HLC 0534568).

We have also investigated the role of prosodic variation in spoken
dialogue systems, focusing on how \TechTerm{given} information is
produced in comparison with \TechTerm{new} information (NSF
IIS-0307905).  In \cite{Hirschberg04} we examined two potential uses
of \TechTerm{downstepped} contours in Standard American English (e.g.,
H* \!H* L- L\% in the {ToBI} system \cite{ToBI92,ToBI94,Beckman04})
which have been hypothesized in the literature, the marking of
discourse structure and the marking of given information.  Examining
the Boston Directions Corpus of read and spontaneous speech
\cite{Hirschberg96a}, we found evidence that downstepped contours,
\Emphasis{do} appear to serve at least these two functions.  In
\cite{Gravano06} we presented complementary findings on how speakers
use pitch, intensity (loudness) and pause to distinguish between given
and new information in the same corpus.  Most interestingly, we found
that speakers' productions of given vs.~new information in both
spontaneous and read speech differ significantly according to whether
the information is realized as a noun or a verb: given verbs are
uttered with higher intensity than new verbs, while the opposite
tendency is observed for nouns.  We also found considerable individual
differences in the realization of given vs.~new information in
general.
  
%%We have also conducted several studies of discourse markers in the
%%Columbia Games Corpus, which is one of the corpora we will examine for
%%the research proposed here.  In \cite{GravanoUP}, we describe a
%%perception experiment designed to test the role of in the
%%interpretation of the word {\it okay}, which can realize many
%%different discourse functions.  We extracted 54 tokens of {\it okay}
%%from the Objects Games portion of the Columbia Games Corpus, and
%%presented them in isolation to 20 paid subjects.  Subjects were asked
%%to make a forced-choice classification of each {\it okay} into {\it
%%  acknowledgment/agreement, back-channel,} or {\it cue beginning
%%  discourse segment} (in short, A, B, or C).  Afterwards, we presented
%%contextualized versions of the same {\it okay} tokens to the subjects,
%%and asked them to make the same forced choices.  In the isolated
%%condition, we found strong correlations between the perception of
%%these functions and acoustic, phonetic and prosodic features. However,
%%in the contextualized condition, these correlations tended to
%%dissappear and to be overridden by contextual features like number of
%%words by each speaker before the target {\it okay}.  The inter-subject
%%agreement was higher than the chance level in both parts of the study,
%%and higher for the contextualized tokens than for the isolated ones.
%%Also, we found evidence suggesting that context helps disambiguate the
%%labeling of C more than it does the labeling of A and B.
%%
%%We are currently conducting a set of experiments
%%\cite{HirschbergUP} intented to find acoustic, phonetic, prosodic,
%%lexical and turn-taking features that act as good predictors of the
%%function of cue-words such as {\it okay, right, yeah,} and {\it
%%  mm-hm}.  Preliminary results show that although each of these words
%%seems to behave differently from the rest, turn-taking features and
%%silence before and after the target words are consistently ranked
%%among the best predictors.

%%% -*-LaTeX-*-
%%% $Id: summary.tex,v 1.9 2005/04/29 22:30:03 martin Exp $
\documentclass[10pt]{report}

%%\pdfoutput=0  %% asks pdflatex to generate DVI output instead of PDF

\ifx\pdfoutput\undefined
  \relax
\else
  \pdfoptionpdfminorversion=2
  \pdfcompresslevel=9
\fi

\usepackage{type1cm}
\usepackage[activate]{pdfcprot}

%% modified ot1ppl.fd, scaled Palatino by 5%
\DeclareFontFamily{OT1}{ppl}{}
\DeclareFontShape{OT1}{ppl}{m}{n} {<-> [1.05] pplr7t}{}
\DeclareFontShape{OT1}{ppl}{m}{sc}{<-> [1.05] pplrc7t}{}
\DeclareFontShape{OT1}{ppl}{m}{sl}{<-> [1.05] pplro7t}{}
\DeclareFontShape{OT1}{ppl}{m}{it}{<-> [1.05] pplri7t}{}

\DeclareFontShape{OT1}{ppl}{b}{n} {<-> [1.05] pplb7t}{}
\DeclareFontShape{OT1}{ppl}{b}{sc}{<-> [1.05] pplbc7t}{}
\DeclareFontShape{OT1}{ppl}{b}{sl}{<-> [1.05] pplbo7t}{}
\DeclareFontShape{OT1}{ppl}{b}{it}{<-> [1.05] pplbi7t}{}

\DeclareFontShape{OT1}{ppl}{bx}{n} {<->ssub * ppl/b/n}{}
\DeclareFontShape{OT1}{ppl}{bx}{sc}{<->ssub * ppl/b/sc}{}
\DeclareFontShape{OT1}{ppl}{bx}{sl}{<->ssub * ppl/b/sl}{}
\DeclareFontShape{OT1}{ppl}{bx}{it}{<->ssub * ppl/b/it}{}

\usepackage{mathpazo}

\usepackage{geometry}
\geometry{margin=2.5 true cm, nohead, nofoot}

\pagestyle{empty}

%\renewcommand{\baselinestretch}{1.05}
\parindent=1em
\hyphenpenalty=0
\widowpenalty=0
\clubpenalty=0

\usepackage[usenames,dvipsnames]{color}
\definecolor{Black}{rgb}{0,0,0}
\ifx\useColor\relax
  \definecolor{Linkcolor}{rgb}{0.7,0.0,0.0}
  \definecolor{Citecolor}{rgb}{0.0,0.5,0.0}
  \definecolor{Filecolor}{rgb}{0.0,0.0,0.5}
\else
  \definecolor{Linkcolor}{rgb}{0,0,0}
  \definecolor{Citecolor}{rgb}{0,0,0}
  \definecolor{Filecolor}{rgb}{0,0,0}
\fi

%%\usepackage{natbib}
%%\bibliographystyle{abbrvnat}
%%\newcommand{\citeg}[1]{\citeauthor{#1}'s \citeyearpar{#1}}

\usepackage[pdfpagelabels]{hyperref}
\hypersetup{
  breaklinks=true,
  plainpages=false,
  colorlinks=true,
  anchorcolor=Black,
  linkcolor=Linkcolor,
  menucolor=Linkcolor,
  pagecolor=Linkcolor,
  citecolor=Citecolor,
  filecolor=Filecolor,
  urlcolor=Filecolor,
  bookmarksopen=true,
  bookmarksnumbered=true,
  pdftitle={Collaborative Research: Translating Prosody in an
    English/Chinese Language Tutoring System},
  pdfauthor={Hirschberg et al.},
  pdfsubject={IIS - Collaborative Systems; IIS - Advanced
    Learning Technologies},
  pdfkeywords={prosody, language training, spoken language systems},
  pdfpagemode=None,
  pdfstartview=FitH
}

\usepackage{hypcap}

\renewcommand{\thechapter}{\Alph{chapter}}
\renewcommand{\thepage}{\Alph{chapter}--\arabic{page}}

\setcounter{secnumdepth}{4}  %% number headings down to subsubsections
\setcounter{tocdepth}{4}     %% and put them in the toc

\makeatletter
\renewcommand{\chapter}{%
  \if@openright\cleardoublepage\else\clearpage\fi%
  \global\@topnum\z@%
  \@afterindentfalse%
  \phantomsection%
  \setcounter{page}{1}%
  \refstepcounter{chapter}%
  %%\addtocontents{lof}{\protect\addvspace{18\p@}}%
  %%\addtocontents{lot}{\protect\addvspace{18\p@}}%
  \secdef\MYchapter\MYsChapter}

\newcommand{\MYchapter}[2][?]{%
  \addcontentsline{toc}{chapter}{\protect\numberline{\thechapter}#1}%
  \vspace*{0pt}%
  {\centering\bfseries\huge\thechapter\quad #2\par}%
  \vspace*{4ex}%
  }
\newcommand{\MYsChapter}[1]{%
  \addcontentsline{toc}{chapter}{\protect\numberline{\thechapter}#1}%
  \vspace*{0pt}%
  {\centering\bfseries\huge\thechapter\quad #1\par}%
  \vspace*{4ex}%
  }
\makeatother


%% local macros

\newcommand{\ChapRef}[1]{\hyperref[{chap:#1}]{Chapter~\ref*{chap:#1}}}
\newcommand{\EqRef}[1]{\hyperref[{eq:#1}]{(\ref*{eq:#1})}}
\newcommand{\FigRef}[1]{\hyperref[{fig:#1}]{Figure~\ref*{fig:#1}}}
\newcommand{\SecRef}[1]{\hyperref[{sec:#1}]{Section~\ref*{sec:#1}}}
\newcommand{\PageLabel}[2]{\hypertarget{p:#1}{\label{p:#1}#2}}
\newcommand{\PageRef}[1]{\hyperlink{p:#1}{page~\pageref*{p:#1}}}

\let\Acronym=\textsc
\let\Concept=\emph

\newcommand{\TechTerm}[1]{{\sc #1}}
\newcommand{\Emphasis}[1]{{\bf \em #1}}
\newcommand{\LingMat}[1]{{\it #1}}

\newcommand{\YN}{\emph{yes\slash no}}

\newcommand{\FIXME}[1]{\textbf{\textcolor{red}{#1}}}

\begin{document}

\setcounter{chapter}{2}

\input{one-page}

%\clearpage
%\bibliography{sp2sp,refs}

\end{document}
\section{List of Senior Personnel}
\ref{sec:pers}
\begin{enumerate}
\item
Julia Hirschberg; Columbia University; PI
\item
Ani Nenkova; University of Pennsylvania; co-PI or PI
\end{enumerate}
