Code it, run it, crash it–restart it.
🔥 the gist
➡️ ES-FoMo 2023 was a blast! Thank you for joining us at ICML in Hawaii, and stay tuned for a 2024 edition.
- what? A workshop to bring together interdisciplinary experts working on the emerging research questions and challenges associated with foundation model training and inference.
- when & where?
- questions? Contact us at
Awards and post-workshop happy hour sponsored by Together.
📆 the plan
All times HST, UTC-10. Find us in Ballroom A (floor L4) of the convention center.
|9:00am||🔥 Session I: Large-Scale Distributed Pretraining|
|Using Megatron to Train Large Language Models||Deepak Narayanan
|Distributed Systems for Decentralized AI||Ce Zhang
|Training Large Language Models on Cerebras Wafer-Scale Clusters AI||Natalia Vassilieva
|10:25am||🎤 Contributed Talk 1|
|SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores||Yi Wu
(Tsinghua University, Shanghai Qi Zhi Institute)
|10:40am||🎤 Contributed Talk 2|
|Finetuning Language Models with Just Forward Passes||Sadhika Malladi
|10:55am||🚀 Session II: Efficient Inference|
|The Case for 4-bit Inference||Tim Dettmers
(University of Washington)
|Efficienly Scaling Transformer Inference||Aakanksha Chowdhery
|11:55am||🎤 Contributed Talk 3|
|Memory-Efficient Selective Finetuning||Antoine Simoulin
|1:00pm||🧑🎓 Poster Session|
|2:15pm||💬 Panel: Large Language Models Tooling Across Industry and Academia|
|Anna Goldie (Anthropic), Rishi Bommasani (Stanford University), Susan Zhang, Emily Webber (AWS), James Bradbury (Google DeepMind)|
|Moderated by Abhi Venigalla (Mosaic.ML), Dylan Patel (SemiAnalysis).|
|3:30pm||🎤 Contributed Talk 5|
|Fast Causal Attention with Dynamic Sparsity||Daniele Paliotta
(University of Geneva)
|3:45pm||⚙️ Session III: Deep Optimization|
|PyTorch 2.x: Faster, More Pythonic, and as Dynamic as Ever||Natalia Gimelshein
|High-Performance Kernel Programming with Triton||Philippe Tillet
|6:00pm||🎉 Post-workshop happy hour||sponsored by Together|
|RSVP on Partiful to get a ticket!|
🦾 the pitch
As models increase in size and training budget, they not only systematically improve in upstream quality, but also exhibit novel emergent capabilities. This increase in scale raises proportionate difficulties for practitioners: foundation model training and inference lie at a unique interdisciplinary crossroad, combining open problems in algorithms, system design, and software engineering.
Machine learning practitioners are key stakeholders here: on the one hand, researchers may contribute algorithmic insights and novel methods to improving training and inference of large models; on the other hand, novel research findings may be best demonstrated at scale—which may require training models as efficiently as possible to make the best use of available resources.
The goal of this workshop is to bring together interdisciplinary experts working on the emerging research questions and challenges associated with foundation model training and inference. We welcome submissions around training and inference systems/algorithms for foundation models, focusing on scaling-up or on reducing compute, time, memory, bandwidth, and energy requirements. Notably, we encourage submissions concerning the entire spectrum of foundation models: from BERT-sized Transformers, to large models with 100B+ parameters. Topics include but are not limited to (see our 📝 call for papers for details):
- Training and inference systems, either distributed at large scale or in resource-constrained scenarios;
- Algorithms for improved training and inference efficiency;
- Systems for foundation models, such as novel programming languages or compilers.