What’s Next in LLM Reasoning? with Roland Memisevic - 646

Today we’re joined by Roland Memisevic, a senior director at Qualcomm AI Research. In our conversation with Roland, we discuss the significance of language in humanlike AI systems and the advantages and limitations of autoregressive models like Transformers in building them. We cover the current and future role of recurrence in LLM reasoning and the significance of improving grounding in AI-including the potential of developing a sense of self in agents. Along the way, we discuss Fitness Ally, a fitness coach trained on a visually grounded large language model, which has served as a platform for Roland’s research into neural reasoning, as well as recent research that explores topics like visual grounding for large language models and state-augmented architectures for AI agents.
🔔 Subscribe to our channel for more great content just like this: ua-cam.com/users/twimlai?sub_confi...
🗣️ CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: twimlai.com/podcast/twimlai/
Join our Slack Community: twimlai.com/community/
Subscribe to our newsletter: twimlai.com/newsletter/
Want to get in touch? Send us a message: twimlai.com/contact/
📖 CHAPTERS
===============================
00:00 - Intro
03:26 - Language as a key ingredient for human-like AI
09:00 - Fitness Alley background
14:00 - Transition from RNNs to attention-based models for better language capabilities
18:42 - GPT-like models lack recurrency; Recurrence can address length generalization
26:09 - Autoregressive models are crucial for building intelligent and agentic AI systems
31:15 - Language is crucial in reasoning; language models lack understanding of code generation but when linked with perceptual input, can improve reasoning capabilities
39:04 - Look, Remember and Reason paper
41:15 - Model architecture; combining language and vision models for reasoning with a top-down approach
46:33 - Combining vision and reasoning develop common sense gaps in AI
50:56 - Situated chat paper
53:51 - Painter paper
01:01:11 - Missing pieces in AI: recurrence, memory, and a sense of self in agents
🔗 LINKS & RESOURCES
===============================
Look, Remember and Reason: Visual Reasoning with Grounded Rationales - arxiv.org/pdf/2306.17778.pdf
Situated Real-time Interaction with a Virtually Embodied Avatar - embodied-ai.org/papers/2023/1...
Painter: Teaching Auto-regressive Language Models to Draw Sketches - arxiv.org/abs/2308.08520
Learning “Common Sense” and Physical Concepts with Roland Memisevic - #111 - twimlai.com/podcast/twimlai/l...
Pixels to Concepts with Backpropagation with Roland Memisevic - #427 - twimlai.com/podcast/twimlai/p...
For a COMPLETE LIST of links and references, head over to twimlai.com/go/646.
📸 Camera: amzn.to/3TQ3zsg
🎙️Microphone: amzn.to/3t5zXeV
🚦Lights: amzn.to/3TQlX49
🎛️ Audio Interface: amzn.to/3TVFAIq
🎚️ Stream Deck: amzn.to/3zzm7F5