“Adaptive and Efficient Computation in Deep Neural Networks”
The continuous growth of machine learning models means that their efficiency is crucial to ensure sustainability and broad access to new technologies. This talk will highlight how emergent modularity in deep neural networks, a property observable across network depth and width, can enable more efficient model inference through adaptive computation. In the talk, I will cover the basic principles of modularity and adaptive computation techniques. Then, I will present my research on dynamic methods for reducing resource usage, such as early-exits and activation sparsity. Finally, I will briefly outline my ongoing work in areas like efficient LLM safety and speculative decoding, and discuss some of the remaining open challenges in achieving efficiency in modern deep learning.
Bio
Filip Szatkowski is a PhD student at the Warsaw University of Technology, where he also earned his MSc in Computer Science. During his PhD, he was supported by IDEAS NCBR. His research focuses on efficiency in deep learning, covering topics such as adaptive computation, activation sparsity, speculative decoding, and continual learning. He has collaborated with researchers from Jagiellonian University, Universitat Autònoma de Barcelona, Sapienza Università di Roma, and the University of Amsterdam, and was a visiting researcher at Sapienza. His work has been published at top conferences including NeurIPS and ICML. Beyond academia, he gained industry experience through internships, developing efficient neural networks at Samsung R&D Warsaw and Amazon AWS AI Tübingen.