Annotation, aggregation, and augury: the lifecycle of labeling from data collection to model prediction

Matthew Lease
Associate Professor in the School of Information
University of Texas at Austin University (NTU)
Professor Yang Liu

Join us on Zoom:

Description: Success in building AI systems involves managing a complex lifecycle of activities spanning data collection to annotation, label agreement and aggregation, effective augury (accurate, fair, and explainable prediction), and ensuring sound evaluation practices. My research adopts a holistic approach to these activities, assuming that the weakest link will limit overall success. To illustrate this range of activities, I present studies drawn from each lifecycle stage. For human labeling, I describe our design of annotator rationales to improve the quality of human relevance judgments in evaluating search results. Next, I describe a general framework for measuring annotator agreement and aggregating labels across a myriad of annotation tasks. Finally, I present our latest work on explainable propaganda detection in news articles, including human-centered evaluation. Future work highlights include balancing fairness vs. accuracy in detecting hate speech, investigating rationale extraction for product reviews under distributional shift, and co-designing useful NLP tools for fact checking with professional fact checkers.

Speaker Bio: Matthew Lease is an Associate Professor in the School of Information at the University of Texas at Austin (UT), with promotion to full Professor as of September 2022. He holds a Ph.D. in Computer Science (CS) from Brown University (August 2009) and a courtesy appointment in UTCS. He has received a number of paper awards, three notable early career awards (NSF, DARPA, and IMLS), and is a faculty founder of UT’s Good Systems ( Grand Challenge to create responsible AI technologies. Beyond UT, Lease works part-time as an Amazon Scholar in Amazon’s human-in-the-loop team.

Lease’s research combines human-centered and system-centered approaches to build quality datasets, create fair and explainable predictive models, design human-in-the-loop systems, and find win-wins for data annotators as well as requesters. His work spans the fields of crowdsourcing / human computation, information retrieval, and natural language processing. A grounding focus of his ongoing research is content moderation, including fact checking, hate speech detection, and image moderation. For more information, please visit his homepage: