Long Context Language Models and their Biological Applications with Eric Nguyen - 690

Today, we're joined by Eric Nguyen, PhD student at Stanford University. In our conversation, we explore his research on long context foundation models and their application to biology particularly Hyena - hazyresearch.s..., and its evolution into Hyena DNA - hazyresearch.s... and Evo - arcinstitute.o... models. We discuss Hyena, a convolutional-based language model developed to tackle the challenges posed by long context lengths in language modeling. We dig into the limitations of transformers in dealing with longer sequences, the motivation for using convolutional models over transformers, its model training and architecture, the role of FFT in computational optimizations, and model explainability in long-sequence convolutions. We also talked about Hyena DNA, a genomic foundation model pre-trained on 1 million tokens, designed to capture long-range dependencies in DNA sequences. Finally, Eric introduces Evo, a 7 billion parameter hybrid model integrating attention layers with Hyena DNA's convolutional framework. We cover generating and designing DNA with language models, hallucinations in DNA models, evaluation benchmarks, the trade-offs between state-of-the-art models, zero-shot versus a few-shot performance, and the exciting potential in areas like CRISPR-Cas gene editing.
🎧 / 🎥 Listen or watch the full episode on our page: twimlai.com/go....
Note: Please note that this episode has audio and video sync issues. We apologize for any inconvenience this may cause and appreciate your understanding. Thank you for watching!
🔔 Subscribe to our channel for more great content just like this: ua-cam.com/users/tw...
🗣️ CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: twimlai.com/po...
Follow us on Twitter: / twimlai
Follow us on LinkedIn: / twimlai
Join our Slack Community: twimlai.com/co...
Subscribe to our newsletter: twimlai.com/ne...
Want to get in touch? Send us a message: twimlai.com/co...
📖 CHAPTERS
===============================
00:00 - Introduction
01:14 - Motivation for Hyena architecture
02:39 - Limitations of transformer architectures with longer sequences
05:06 - Role of Fast Fourier Transform (FFT) in Hyena
07:54 - Explainability in long-sequence convolutions
09:07 - Hyena model
14:45 - Hyena DNA
19:10 - Hyena DNA model training
21:11 - Evo
24:32 - Designing DNA with language models
25:52 - Transformer-based approaches to DNA
28:21 - Hallucination in DNA models
33:41 - Evo gene editing tools
35:30 - Evo evaluation benchmarks
38:21 - Evo vs state-of-the-art models
40:38 - Zero-shot vs a few-shot performance
42:06 - Future directions
🔗 LINKS & RESOURCES
===============================
Hyena Hierarchy: Towards Larger Convolutional Language Models - hazyresearch.s...
HyenaDNA: learning from DNA with 1 Million token context - hazyresearch.s...
Evo: DNA foundation modeling from molecular to genome scale - arcinstitute.o...
📸 Camera: amzn.to/3TQ3zsg
🎙️Microphone: amzn.to/3t5zXeV
🚦Lights: amzn.to/3TQlX49
🎛️ Audio Interface: amzn.to/3TVFAIq
🎚️ Stream Deck: amzn.to/3zzm7F5