Towards a Theoretical Understanding of Parameter-Efficient Fine-Tuning (and Beyond)

Kangwook Lee
Assistant Professor
Electrical and Computer Engineering and Computer Sciences Department
University of Wisconsin-Madison
Abstract: As pretrained models grow in complexity and size, traditional transfer learning methods, which adjust all the model parameters, are becoming increasingly expensive. In response, parameter-efficient fine-tuning, a new adaptation method that updates only a tiny fraction of the model parameters, leaving the remainder unchanged, has gained popularity. Despite the growing importance of this emerging paradigm, its theoretical foundations remain largely unexplored. Current algorithms and design choices substantially rely on heuristics, signaling a significant opportunity for a comprehensive, theoretical exploration of this novel paradigm. I will introduce two recent results in this new area: the expressive power of low-rank adaptation (also known as LoRA), and the memorization capacity of side-tuning. I will conclude the talk by briefly discussing prompting and looping, a completely different approach to adaptation that does not involve any parameter updates.
Bio: Kangwook Lee is an Assistant Professor at the Electrical and Computer Engineering Department and the Computer Sciences Department (by courtesy) at the University of Wisconsin-Madison. Previously, he was a Research Assistant Professor at the Information and Electronics Research Institute of KAIST and was a postdoctoral scholar at the same institute. He received his PhD in 2016 from the Electrical Engineering and Computer Science department at UC Berkeley. He is the recipient of The IEEE Joint Communications Society/Information Theory Society Paper Award (2020) and the KSEA Young Investigator Grants Award (2022).

[@maktube_220]

great presentation!!

[@Mome3600]

Thanks ! :D The LIFT idea is absolutely amazing !((: