CSH Talk by Shujian Yu “Information Theory meets Deep Learning”

Shujian Yu from UiT – the Arctic University of Norway will present a talk on Friday, July 21st, 2023, at 3 PM in the Salon.

Title: Information Theory meets Deep Learning

Abstract: In the era of data deluge, various deep learning algorithms have been proven extraordinarily successful in providing tools and paradigms for extracting information from massive data sets. Despite this success, relatively less attention has been paid to these algorithms’ fundamental limits and trade-offs. Information theory demonstrates excellent potential in this context. To bridge the gap between traditional information theory and real-world deep learning problems, we propose the matrix-based Renyi’s entropy functional from a sample-level, which evaluates information-theoretic quantities on the eigen-spectrum of symmetric positive definite (SPD) matrices without any density estimation and distributional assumptions. Based on the new entropy function, we demonstrate how information theory can be used to improve the generalization performances of deep neural networks. In this talk, we focus our discussions on two learning principles: 1) the minimum error entropy; and 2) the deep “deterministic” information bottleneck.

Bio: Shujian Yu is currently an assistant professor at the Department of Computer Science of the Vrije Universiteit Amsterdam. He also holds an associate professor II position in the Machine Learning Group at the UiT – The Arctic University of Norway. He received his Ph.D. degree in Electrical and Computer Engineering from the University of Florida in 2019, with a Ph.D. minor in Statistics. His research interests include machine learning, information theory, and signal processing. He received the 2020 International Neural Networks Society (INNS) Aharon Katzir Young Investigator Award. He was also selected for the 2023 AAAI New Faculty Highlights.