Powering AI with the World's Largest Computer Chip with Joel Hestness - 684

Today we're joined by Joel Hestness, principal research scientist and lead of the core machine learning team at Cerebras. We discuss Cerebras’ custom silicon for machine learning, Wafer Scale Engine 3, and how the latest version of the company’s single-chip platform for ML has evolved to support large language models. Joel shares how WSE3 differs from other AI hardware solutions, such as GPUs, TPUs, and AWS’ Inferentia, and talks through the homogenous design of the WSE chip and its memory architecture. We discuss software support for the platform, including support by open source ML frameworks like Pytorch, and support for different types of transformer-based models. Finally, Joel shares some of the research his team is pursuing to take advantage of the hardware's unique characteristics, including weight-sparse training, optimizers that leverage higher-order statistics, and more.
🔔 Subscribe to our channel for more great content just like this: ua-cam.com/users/twimlai?sub_confi...
🗣️ CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: twimlai.com/podcast/twimlai/
Follow us on Twitter: / twimlai
Follow us on LinkedIn: / twimlai
Join our Slack Community: twimlai.com/community/
Subscribe to our newsletter: twimlai.com/newsletter/
Want to get in touch? Send us a message: twimlai.com/contact/
📖 CHAPTERS
===============================
00:00 - Introduction
03:00 - Cerebras and AI hardware
18:32 - Cerebras vs alternatives such as GPUs, TPUs, and AWS Inferentia
21:28 - How does the WSE work?
33:56 - Who benefits from such a device?
47:31 - Building hardware around transformers
52:07 - Benefits of Cerebras research
54:38 - Conclusion
🔗 LINKS & RESOURCES
===============================
nanoGPT - github.com/karpathy/nanoGPT
Cerebras - www.cerebras.net/
TensorFlow - www.tensorflow.org/
PyTorch - pytorch.org/
JAX - jax.readthedocs.io/en/latest/...
AWS Inferentia - aws.amazon.com/machine-learni...
TPU - cloud.google.com/tpu
📸 Camera: amzn.to/3TQ3zsg
🎙️Microphone: amzn.to/3t5zXeV
🚦Lights: amzn.to/3TQlX49
🎛️ Audio Interface: amzn.to/3TVFAIq
🎚️ Stream Deck: amzn.to/3zzm7F5