Reinforcement and Adversarial Learning for Situated Interactive Systems

Ian Lane
Associate Professor Electrical and Computer Engineering in AI Systems
Carnegie Mellon
Professor Xin Wang

Join us on Zoom:

Description: Despite recent advances in machine learning, speech recognition and natural language processing, understanding and interacting with users in physically situated environments still remain a significant challenge. In order for interactive systems to converse with users in a situationally-aware manner, they must take into account not only what the user has said but also their gestures, actions, and the spatial and temporal context within the environment. In this talk, I will introduce recent work on adversarial and reinforcement learning for end-to-end spoken dialog systems, and the extension of these models to situated environments in which a system selects a physical action or natural language response taking into account the users input, the environment, and the situational context.

In this talk, I will introduce neural-network-based dialog systems and present approaches for supervised, reinforcement, and adversarial learning of these models. I will compare the performance and learning efficiency of each method for task-orientated dialog and will present our work applying these models in situated environments, focusing on the task of vision-grounded instruction following. I will discuss how the presented approaches can be applied to other situated tasks and the challenges that need to be overcome to realize natural, situated interaction for the conversational AI systems of the future.

Speaker Bio: Ian Lane is the Sense of Wonder Group Associate Professor of Electrical and Computer Engineering in AI Systems at Carnegie Mellon University, and an affiliated faculty at CMU’s Language Technology Institute. Prior to joining CMU, he was a scientist at the Advanced Telecommunications Research Institute in Kyoto, Japan where he performed research on speech recognition and spoken language understanding while completing his PhD at Kyoto University. His research focuses on spoken language technologies and machine learning, including speech recognition, spoken language translation, spoken dialog systems, and situated interaction. At CMU, Ian co-founded the NVidia Teaching Center, the CUDA Center of Excellence, and the College of Engineering research cluster at the Pittsburgh Supercomputing Center. He is the inaugural recipient of the Sense of Wonder Group Professorship at CMU and has received faculty research awards from Amazon, Nvidia, and Google.