Deep Reinforcement Learning for Fluid Dynamics and Control
Вставка
- Опубліковано 10 чер 2024
- Reinforcement learning based on deep learning is currently being used for impressive control of fluid dynamic systems. This video will describe recent advances, including for mimicking the behavior of birds and fish, for turbulence closure modeling with sub-grid-scale models, and for robotic flight demonstrations.
Citable link for this video: doi.org/10.52843/cassyni.kvtnvy
@eigensteve on Twitter
eigensteve.com
databookuw.com
Links to papers in video:
@3:58 Machine learning for fluid mechanics
Brunton, Noack, Koumoutsakos, Ann. Rev. Fluid Mech 52:477--508, 2020
www.annualreviews.org/doi/pdf...
@5:04 Efficient collective swimming by harnessing vortices through deep reinforcement learning
Verma, Novati, Koumoutsakos, Proc. Nat. Acad. Sci. 115(23):5849--5854, 2018
www.pnas.org/content/115/23/5849
@6:57 Automating turbulence modelling by multi-agent reinforcement learning
Novati, Lascombes de Laroussilhe, Koumoutsakos, Nat. Mach. Int. 3:87--96, 2021
www.nature.com/articles/s4225...
@8:47 A review of Deep Reinforcement Learning for fluid mechanics,
Garnier, Viquerat, Rabault, Larcher, Kuhnle, Hachem, 2019
arxiv.org/abs/1908.04127
@9:57 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
Rabault, Kuchta, Jensen, Reglade, Cerardi, J. Fluid Mech. 865, 2019
doi.org/10.1017/jfm.2019.62
@10:56 Reinforcement learning for bluff body active flow control in experiments and simulations
Fan, Yang, Wang, Triantafyllou, Karniadakis, Proc. Nat. Acad. Sci. 117(42), 2020
doi.org/10.1073/pnas.2004939117
@11:50 Fluid directed rigid body control using deep reinforcement learning
Ma, Tian, Pan, Ren, Manocha, SIGGRAPH 2018
gamma.cs.unc.edu/DRL_FluidRigid/
@13:26 Autonomous helicopter flight via Reinforcement Learning
Ng, Kim, Jordan, Sastry, NeurIPS 2004
papers.nips.cc/paper/2003/fil...
@13:26 An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, Coates, Quigly, Ng, NeurIPS 2007
proceedings.neurips.cc/paper/...
@13:26 Autonomous helicopter aerobatics through apprenticeship learning
Abeel, Coates, Ng, Int J Rob Res 2010
journals.sagepub.com/doi/abs/...
@14:02 Learning to fly like a bird
Tedrake, Jackowski, Cory, Roberts, Hoburg, Int. Symp. Rob. Res. 2009
groups.csail.mit.edu/robotics...
@14:58 Control of a Quadrotor with Reinforcement Learning
Hwangbo, Sa, Siegwart, Hutter, IEEE Rob Aut 2(4) 2017
arxiv.org/abs/1707.05110
@15:22 Learning to soar in turbulent environments
Reddy, Celani, Sejnowski, Vergassola Proc. Nat. Acad. Sci. 113(33, 2016
www.pnas.org/content/113/33/E...
@16:31 Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning
Xu, Du, Foshey, Li, Zhu, Schulz, Matusik, SIGGRAPH 2019
people.csail.mit.edu/jiex/pap...
This video was produced at the University of Washington - Наука та технологія
Links to papers in video:
@3:58 Machine learning for fluid mechanics
Brunton, Noack, Koumoutsakos, Ann. Rev. Fluid Mech 52:477--508, 2020
www.annualreviews.org/doi/pdf/10.1146/annurev-fluid-010719-060214
@5:04 Efficient collective swimming by harnessing vortices through deep reinforcement learning
Verma, Novati, Koumoutsakos, Proc. Nat. Acad. Sci. 115(23):5849--5854, 2018
www.pnas.org/content/115/23/5849
@6:57 Automating turbulence modelling by multi-agent reinforcement learning
Novati, Lascombes de Laroussilhe, Koumoutsakos, Nat. Mach. Int. 3:87--96, 2021
www.nature.com/articles/s42256-020-00272-0
@8:47 A review of Deep Reinforcement Learning for fluid mechanics,
Garnier, Viquerat, Rabault, Larcher, Kuhnle, Hachem, 2019
arxiv.org/abs/1908.04127
@9:57 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
Rabault, Kuchta, Jensen, Reglade, Cerardi, J. Fluid Mech. 865, 2019
doi.org/10.1017/jfm.2019.62
@10:56 Reinforcement learning for bluff body active flow control in experiments and simulations
Fan, Yang, Wang, Triantafyllou, Karniadakis, Proc. Nat. Acad. Sci. 117(42), 2020
doi.org/10.1073/pnas.2004939117
@11:50 Fluid directed rigid body control using deep reinforcement learning
Ma, Tian, Pan, Ren, Manocha, SIGGRAPH 2018
gamma.cs.unc.edu/DRL_FluidRigid/
@13:26 Autonomous helicopter flight via Reinforcement Learning
Ng, Kim, Jordan, Sastry, NeurIPS 2004
papers.nips.cc/paper/2003/file/b427426b8acd2c2e53827970f2c2f526-Paper.pdf
@13:26 An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, Coates, Quigly, Ng, NeurIPS 2007
proceedings.neurips.cc/paper/2006/file/98c39996bf1543e974747a2549b3107c-Paper.pdf
@13:26 Autonomous helicopter aerobatics through apprenticeship learning
Abeel, Coates, Ng, Int J Rob Res 2010
journals.sagepub.com/doi/abs/10.1177/0278364910371999
@14:02 Learning to fly like a bird
Tedrake, Jackowski, Cory, Roberts, Hoburg, Int. Symp. Rob. Res. 2009
groups.csail.mit.edu/robotics-center/public_papers/Tedrake09.pdf
@14:58 Control of a Quadrotor with Reinforcement Learning
Hwangbo, Sa, Siegwart, Hutter, IEEE Rob Aut 2(4) 2017
arxiv.org/abs/1707.05110
@15:22 Learning to soar in turbulent environments
Reddy, Celani, Sejnowski, Vergassola Proc. Nat. Acad. Sci. 113(33, 2016
www.pnas.org/content/113/33/E4877
@16:31 Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning
Xu, Du, Foshey, Li, Zhu, Schulz, Matusik, SIGGRAPH 2019
people.csail.mit.edu/jiex/papers/LearningToFly/index.html
You mentioned that your wife is a neuroscientist and indicated that animal reward is internal. Can you by chance provide a reference article that would describe this in greater detail?
It is so pleasant to see someone doing their work so passionately! You are an outstanding professor, Dr Brunton!
We need more professors and lecturers like you Dr. Brunton. You made academic publications more interesting to study !
This channel is pure gold.
Thanks!
the best teacher I havve ever seen on youtube . greetings and regards from INDIA !!!!
Wow, thank you!
I really enjoy the enthusiasm you show when delivering the topics! Perfect and outstanding...
Thank you very much for sharing you great papers and knowledge!
My pleasure!
Thank you professor, it's great to watch your videos!
Glad you like them!
I'm thinking about Jim Lovell and Apollo 13. After the explosion, he had to relearn flying the craft in a new configuration. He said it would go left when he wanted to go right. But they did it. This kind of work could save lives when we suddenly find ourselves in a new place we didn't count on.
I love these kinds of examples, where humans are able to rapidly re-learn on the fly.
Great video! One thing I'm curious about - my understanding is that reinforcement learning is difficult in practice b/c it's hard to build a simulated environment which matches reality. But here, it seems that issue has been managed, since these drones are flown in the real circumstance. So, are our turbulence simulations really that good or are there other clever tricks here? In general, do you see the accuracy of the simulation as the primary limiting factor in reinforcement learning?
The legend of control engineers 🙏 thanx
Thanks!
Hey Steve, thanks for the great video and the paper highlight ;)
It makes me want to join Washington University
I'm enrolled in prof. Brunton's April offering at UW, "machine learning for fluids". Will let you know how it is
@@harv609 is this offered as an online course?
Awesome, apply to our program in mechanical engineering!
@@kevalan1042 most courses at UW are offered remotely because of covid, so yea. It's offered as an online course this term
@@Eigensteve are you planning to offer the course on a MOOC platform such as Coursera or edX?
Thank you, fantastic video!
Glad you liked it!
Using the thermals to climb reminds me of the finite horizon, energy optimal trajectory video you just posted.
Dear prof,
I really admire it.. Your videos make to do research on AI (DRL) in CFD
Thank you professor.
Thank you for making this video
My pleasure!
a great video! Thank you
Glad you liked it!
Great video and content!
Would be really nice if you share how you are creating this video with content overlayed on screen.
Amazing video thank you
Thank you too!
Fantastic job big thanks
Thank you too!
This makes me really want to get into UW's CS program. Fingers crossed!
Fantastic Video
please put the links of videos and papers you mentioned!
Great point. Just pinned a comment with links to all papers referenced.
This is so cool !
RL fort the win !
Dear Sir, thank you for your channel and your book. These are a real treasure trove of very useful tools for the ‘honest mind’ of the 21st century. I was wondering whether you could direct me to resources regarding genetic algorithms. I have a couple of finance related optimisation problems involving discrete variables (essentially inclusion or exclusion in a portfolio subject to certain constraints) and I believe the genetic algorithms would be the most suitable in these cases. I would be extremely grateful for any guidance/suggestion on this. Many thanks and best regards from sunny Singapore.
Question: Are the fluid equations solved using a deep learning, or is it done through more traditional solvers like finite element?
here we go!
So these simulations (like in the case of the fish) are essentially used as RL environments, correct? Are these used within the framework of gym or what kind of toolset is used?
Well done!
Thanks!
So in the case of biological systems the reward is actually just a function of the state ?
Well it is somehow internally motivated, since it is your own body that is dumping the dopamine.
Perfectly 🎉
Do you think these AI control techniques will replace current robust controller design techniques such as mu-synthesis, etc.?
I guess the advantage of traditional methodologies is that they come with mathematically proven guarantees on stability and performance, though no doubt they have disadvantages, too.
I don't think so, but they serve different purposes. Eventually I see RL leveraging more traditional control techniques more effectively.
This is very very cool. Thanks for shooting this video
Steve, have you ever thought about creating a MOOC?
I have, but I like keeping everything 100% open on UA-cam. But check out databookuw.com for more information on syllabi, homework, etc.
Nice
Thanks
Could u talking about architecture robot interactive/creative and AI
At 2:47 you said that in biological systems, the reward comes in the form of dopamine. But how is that dopamine released in the body? Wouldn't it be according to the environment we are in (or when we tell our mind that we are in a positive environment)?. Let me know.
Yes, but it is delivered internally, based on our internal perception of the external state.
@@Eigensteve but isn’t that equivalent to our brain thinking that we’ve got some positive response in the environment?
Dear professor please prepare an education about using Q learning PID contriller
11:42
yeah, those people are called pitchers ...
Hah, good call...
@@Eigensteve steeeeeRiiiiiik
Probably SpaceX uses RL to land their rockets in the ships. It would be very nice to know something about that, however it should be secret.
Interesting... secret RL!
Pretty sure they use classical control. RL doesn't make a lot of sense in that particular case
@@seremetvlad could you explain why you think it doesn't make sense? I'm curious to understand
e-fish-ently, i got it
Nice :)
Don't worry, we'll find the mystery behind this 'one person' solved by tomorrow 10'am.
3 unavailable videos are hidden?
Doctor Brunton, so these 3 videos are still not open to public? But I still appreciate your first 5 videos which helps me understand reinforcement learning much more deeply. Thanks!