Time: 1245pm-145pm, Friday, March 19
Place: Room 6496, CUNY Graduate Center, 365 Fifth Ave (34str&35str).
Speaker: Paul Tepper (Northwestern)
Title: Modeling Iconic Gesture Generation in Humans and Virtual Humans


When expressing information about spatial domains, it is natural for
people to accompany their speech with gestures, and more specifically,
iconic gestures, which express visual and spatial information about
objects and actions. Virtual humans and embodied conversational agents
(ECAs) can produce these kinds of gestures to improve on the naturalness
and clarity of their communication in human-computer dialogue. However,
unlike people, these systems typically rely on a library of pre-scripted
or "canned" gestures.

In this talk, I will discuss my research group’s work on modeling
generation of novel iconic gestures in coordination with natural
language. I will start by presenting our framework for analyzing
gestural images into semantic units (image description features), and
linking those units to features of gesture form, or morphology (e.g.
hand shape and movement trajectory). I will describe our application of
this theory in a study of people giving directions around a college
campus. This will also include discussion of the use of perspective in
gestures that indicate the location of landmarks.

Then I will review the NUMACK system, wherein this framework has been
used to implement an ECA with a multimodal microplanner. This system
derives the form of both language and gesture directly from a common set
of communicative goals, enabling generation of novel gestures and
coordinated language on-the-fly. Finally, I will present a new, ongoing
study aimed at extending this line of research with a more
domain-general approach, based on lessons learned from the work on

Speaker Bio:

Paul Tepper is a Ph.D. candidate in the Technology and Social Behavior
program, a joint Ph.D. from the School of Communication and Robert E.
McCormick School of Engineering and Applied Science at Northwestern
University. Paul’s research focuses on developing computational models
of face-to-face conversation, implemented in Embodied Conversational
Agents (ECAs) - virtual humans capable of communicating using language
and non-verbal behavior. An intrinsically interdisciplinary project,
this work also involves the collection and analysis of empirical data to
inform and motivate the design of these models and their implementation.
His current work is on modeling generation of coordinated language and
iconic gestures, based on careful study and analysis of people using
visual and spatial language and gestures, in tasks such as giving
directions and describing 3D shapes. His previous research includes the
use of ECAs for modeling rapport, common ground and interpersonal
relationships, based on theories and evidence about how people develop
and show rapport in interpersonal communication. In 2003, Paul completed
an M.Sc. in Artificial Intelligence, specializing in Human Language
Technology at the University of Edinburgh. He also holds a bachelor’s in
Computer Science, Cognitive Science and Linguistics from Rutgers