Role Of Shannon Entropy As A Regularizer Of DeepNNs With Prof Jose Dolz @ETS Montreal

Abstract:
With the advent of deep learning models a variety of additional terms have been integrated into the main learning objective, which typically serve as a regularizer of the model predictions. This is the case, for example, of the Shannon entropy, which has been widely used in semi-supervised learning to penalize high-entropy predictions, and therefore encourage confident predictions on the unlabeled samples. Nevertheless, having this term as the only learning function is not sufficient, as it obtains its minimum when all data points are assigned to the same class, typically yielding trivial solutions. To overcome this limitation, many recent works have coupled this term with a strong prior, which guides the entropy term, and avoids the model to converge towards such trivial solutions. In this talk, several relevant works where the Shannon entropy is coupled with other learning objectives during training, as well as in testing, will be presented, showing that minimizing the entropy on the predictions has the potential to provide state-of-the-art performances in a variety of learning scenarios.
Speaker Bio:
Jose Dolz is currently Associate Professor at ETS Montreal. His current research focuses on deep learning, medical imaging, optimization and learning strategies with limited supervision. He authored over 60 fully peer-reviewed papers, many of which published in the top venues in medical imaging (MICCAI/ MedIA/TMI/IPMI/NeuroImage), vision (CVPR) and learning (ICML, NeurIPS). His recent work strongly aligns with the content of this tutorial, including few-shot learning, weakly and semi-supervised segmentation, and unsupervised learning. His recent research works in image segmentation achieved top-rank positions in several international medical image segmentation challenges: MICCAI 2017 iSeg, MRBrainS and MICCAI 2021 ENIGMA. Furthermore, some of his works received other recognitions, such the MIDL 2021 best-paper award.