Evaluation of Content in Automated Speech Assessment

Keelan Evanini (ETS)

With the continued rise of English as a global language in industry and academia,
it is becoming increasingly important to provide automated assessments of English
that evaluate the speaking proficiency of a wide range of non-native speakers. In
order to provide a valid assessment of all aspects of speaking proficiency that 
are necessary components of communicative competence, the automated assessment 
needs to be able to evaluate the content of the speech in addition to other 
components that are easier to evaluate by automated speech processing technology, 
such as fluency.  This presentation will first describe the architecture of 
SpeechRater, a capability for automated speech assessment developed at 
Educational Testing Service, and then present recent research into several 
different approaches to automated content evaluation for large-scale assessments
of non-native speech.  Several different speaking tasks types will be 
investigated, ranging from restricted speech (e.g., reading a paragraph out 
loud) to spontaneous speech (e.g., presenting an opinion about a given topic), 
and the performance of the features will be evaluated through correlations 
with proficiency scores provided by human raters.

Keelan Evanini is a Managing Research Scientist in the Research & Development 
division at Educational Testing Service in Princeton, NJ. As a member of the 
Speech and Natural Language Processing group, he has conducted research into 
automated spoken language assessment for a wide variety of assessment 
products. Some of his main research interests while at ETS have included 
developing linguistically informed methods of assessing non-native 
pronunciation and prosody, automated content assessment for non-native spoken
responses, and using spoken dialog systems for language learning and 
assessment. He also manages the Research Engineering team which provides 
programming support for NLP / Speech research scientists.  Keelan received his
Ph.D. in Linguistics from the University of Pennsylvania in 2009, and has been
at ETS since then.