How AI Upscaling Improves Weather Forecasts
Вставка
- Опубліковано 30 січ 2023
- How can machine learning improve weather forecasts? Use NordVPN to improve your browsing experience! www.NordVPN.com/simonclark
A paper last year demonstrated that machine learning techniques similar to image upscaling can improve the forecast of the weather, in particular notoriously difficult-to-predict precipitation. In this video I talk to two of the paper's authors about their Generative Adversarial Network (GAN) approach and how they adapted it to the chaotic system of the atmosphere.
You can support the channel by becoming a patron at / simonoxfphys
The paper: agupubs.onlinelibrary.wiley.c...
--------- II ---------
More about me www.simonoxfphys.com/
My second channel - / simonclarkerrata
Twitter - / simonoxfphys
Insta - / simonoxfphys
Twitch - / drsimonclark
--------- II ---------
Music by Epidemic Sound: epidemicsound.com
Some stock footage courtesy of Getty.
Edited by Luke Negus.
How is the weather predicted? Why are weather forecasts wrong? How to predict rain accurately? Can you use machine learning to predict the weatehr? This video is about a scientific paper that blends machine learning techniques with weather prediction, using a generative adversarial network to upscale (downsample) predictions of rainfall over the UK. If you like videos from Jordan Harrod, Veritasium, or Real Engineering you will enjoy this science video about predicting the weather.
Huge thanks to my supporters on Patreon:
Mark Moore, Philipp Legner, Zoey O'Neill, Veronica Castello-Vooght, Jan Krüger, Heijde, Liat Khitman, Michael Parmenter, Matthew Powell, Stormchaser007 , Daan Sneep, Abou El-Ela the Physics Boi, Cody VanZandt, bitreign33 , Andy Hartley, Lachlan Woods.
Chaotic Brain Person, Simon H., Julian Mendiola, Woufff, Ben Cooper, Mark Injerd, dryfrog, Justin Warren, Jack Grimm, Angela Flierman, Alipasha Sadri, Calum Storey, Mattophobia, Riz, The Confusled, Conor Safbom, Paul H and Linda L, Simon Stelling, Gabriele Siino, Bjorn Bakker, Ieuan Williams, Candace H, Tom Malcolm, Leonard Neamtu, Brady Johnston, Kent & Krista Halloran, Rapssack, Kevin O'Connor, Timo Kerremans, Thomas Rintoul, Lars Hubacher, Ashley Wilkins, Samuel Baumgartner, Dan Sherman, ST0RMW1NG 1, Adrian Sand, Morten Engsvang, Farsight101, Haris Karimjee, K.L, fourthdwarf, Felix Freiberger, Chris Field, ChemMentat, Kolbrandr, , Sebastain Graf, Dan Nelson, Shane O'Brien, Alex, Fujia Li, Jesper Koed, Jonathan Craske, Albrecht Striffler, Igor Francetic, Jack Troup, HandsomeCaveman, Sven Ebel, Sean Richards, Kedar , Omar Miranda, Alastair Fortune, Mat Allen, Rafaela Corrêa Pereira, Colin J. Brown, Mach_D, Thusto , Keegan Amrine, Dan Hanvey, Simon Donkers, Kodzo , James Bridges, Liam , Andrea De Mezzo, Wendover Productions, Kendra Johnson. - Наука та технологія
Look mom, Simon is a real youtuber, he got the NordVPN sponsorship!
no a real UA-camr gets sponsored by "raid shadow legends!"
No, a real youtuber suffers with investment scams here in the replies! 🤪
A real UA-camr takes one video to explain their burnout
Given whats happening in Auckland NZ now and the past couple of days this is highly relevant and timely
in case of Auckland the rainfall forecast like 1 to 2 weeks ahead would not help. There's a lot of discussion about their drain infrastructure and general urbanisation. This time nobody can't blame forecast, but more infrastructure preparation... and also crisis case preparation...
Y'all remember when billionaires were huffing on that copium? "Just move to New Zealand! Climate change can't get you if you're down there."
6:35 what the network does is still a very big question mark. training the model is analogous to linear regression, but what those weights do is a much more complicated picture.
I hope we'll not lose the science and the ingenuity that went behind the models, as dependent on initial conditions as they are.
Just as generating a torrent of A.I. art can stop people from going into Art, learning techniques, lighting, composition, perspective, etc., I wish we will keep improving the principles that could let us carry on if global supply chains are not able to manifacture the insanely complicated tech needed to sustain our current system (if for example, fossile fuels came to be costlier and costlier to extract, refine, process, transport, etc.).
Let's not surrender everything to A.I., though it's a great tool to use while we can.
Modern physics-based weather forecasts are also run on purpose-built supercomputers.
@MeOnStuff True. That's not my gripe though.
Personally, I don't think using AI for art and science will make art and science less popular any more than the onset of recorded music made playing instruments less popular - giving people new tools to use has the potential to compound their interest, and make the field more accessible! Another analogy could be calculators. Someone may justified in worrying that people are worse at basic arithmetic, but I believe the possibilities that the tech unlocks are well worth it.
But of course, it's good to have some skepticism and not blindly accept all technology. Your concerns are good points and valid things to keep an eye on!
Currently, AI is very good at taking the average out of a huge pile of data, but it cannot innovate on that by itself. Because of that, there are still many aspects of human society and existence that cannot (yet) be replaced by artificial intelligence. It can, however, make those things easier by sorting through ridiculous amounts of data in a much shorter time and without getting depressed doing it, which makes it very useful indeed, but it won't replace everything just yet. If ever.
This is such a good video. You explained complicated ideas in very digestible terms, and the interviews were awesome! Keep it up!
This is very similar to what I am researching: Species distribution models. Many of the terms are the same, but instead of weather forecast, we do species forecasts.
It was cool to learn about a use for GAN that isn't just bypassing licensing and copyright laws. An area close to where I live got hit by desasterous rain and floods two years ago and while there where warnings, nobody knew how bad it would get. It would be great to get more accurate forecasts.
One thing that’s easier to predict than precipitation: every machine learning researcher watching this video reeling from the sick linear regression burn
Love videos like this where you dive into a specific paper. Super helpful for me as I try to pick a graduate project in atmospheric and oceanic sciences!
Good luck with your graduate project!
thanks for another brilliant video and congrats on getting the sponsorship. thumbs up.
👍
I get so mad when people criticize meteorology when meteorologists and atmospheric scientists get short term weather wrong. Our atmosphere is extremely complex and it takes a ton of physics to figure it out. The good news is predictions are getting more accurate and I wish people had more confidence in science. Thank you for this, atmospheric science is extremely interesting to me.
It's funny how I knew all my life that it doesn't rain without clouds yet never cottoned onto the idea that a raincloud can be statistically microscopic on the macroscopy of a weather forecast. I'd never understand the brilliance of this neat trick if you didn't put that out there.
Really happy that Nord is sponsoring your video. They're Tom Scott approved, a man miserly with his endorsements.
Why is it a neat trick?
Dr Andrew and Dr Lucy are showing us an example of uppers and downers :D
Hard to concentrate on one, and hard to stay interested in the other >
Thank you! I have loved Chaos Theory for years (brother is a math PhD, I'm an engineer) and I was hoping to see one.
as someone who lives in ireland this is somewhat concerning, but luckily it wasn't too bad where I lived
Hi Simon, I've read your book and its ok. It appealed to my "who did that and how" outlook.
In this vid you appear to be telling us that when you come up with a new model its a good idea to see what result it would have predicted on past events with past data. Good sense really.
Really interesting video. Enjoyed this a lot
I’ve been working on AI-based upscaling a fair bit over the last year, so this is pretty interesting. Great to see another real-world use case.
That was great Simon! Really interesting.
I noticed about 5 minutes into the video that there was some plinky piano background music... umm....
It's funny when you got to the sponsor spot and the way you announced it, I was guessing it was either Raid Shadow Legends or Nord VPN
lol congrats on the nord VPN sponsorship
Great video, love a bit of weather.
I've been a fan of yours since 2017, love the content
Sick thumbnail, man. Loved it.
This meteorologist appreciates this video!
Amazing video
I’m glad I’m studying the PBL, I think we have a bit more work to do on our parameterizations there.. interesting video though, I hope a benefit of these “AI” models will be allowing for more hi-res nwp due to the freed up computational resources.
Love the video Simon. One serious question, where did you get the jumper? Looks great.
TV's predicts what pixels will most likely be next to the pixel it has next to it, in this way it needs much less data to show a picture. That is why some times you can see lags in the colors if the colors are very similar.
1:10 had me proper cracked up 😅 ... and with fond memories of Scotland. Where it actually snows sometimes! But not much on the coast, where I was. So ... rain, yeah.
fabulous - the techs living in basement flats in New York City a few weeks ago would have appreciated the forecast as their flat went under water
I really hope that during my lifetime we will still always double-check the weather forecasts with our existing models. Machine learning can make a big poopoo when you least expect it, in a way traditional phyiscs simply can't.
5:41 Quick correction, neural network’s whole purpose is to solve NON linear problems, thus perform non linear calculations
It's funny you're talking about super resolution as it's one of my research subjects atm. I'm doing super resolution of 3D Large Eddy Simulation data using a machine learning algorithm
How likely is it, that this video gets uploaded the day before I start my PhD in meteorology, where a large part of my work will be with post Processing 😅
Hello Simon. This is actually quite a great video. Has there been some applications on Hydrology and water resources? And could it be used for predicting other climatic variables especially those with implications in disaster risk management?
I think this is a really insightful question. I have now left Oxford university, but before I left I was working on applying this technique to improve hydrological models. The other members of the group are currently working on exactly what you've mentioned here. We all think it is really important that the science is applicable to the real world.
@@lucyharris2127 I'd honestly love to learn and try applying it in the country that I reside. Is it possible to be guided until I'm able to navigate through. I'm early career researcher with little experience, so that would be helpful.
Please share links to some research articles too.
@@ofentsewenalesego822 that sounds like a great project - you will have insights from living in that country to develop the model that no one else will. In terms of guidance, I'm still early in my career myself but I'm always happy to chat and help where I can!
@@lucyharris2127 great. Just sent you a connection request on another platform. We can chat from there.
Honestly one of the places that would probably appreciate this the most is the national/state Department of Transportation. Having a better idea where precipitation will fall would massively help allocate resources more quickly and effectively.
Thank you Simon for a wonderful video! I'm strongly considering sending this video to a State Climatologist i've worked with🤣
"It's really just linear regession" 😄
Yessir it is...but don't tell the funder
I've been wondering why we were so delayed in being able to buy Simon's book in Canada (I got a copy at Christmas) and why the soft cover is also delayed here. What's the reasoning? I'm sure there is one I just don't understand why we had to wait for it here. Cheers from Toronto
It snowed in las vegas last night and almost every mountain around las vegas is covered in snow.
Another reason we don't see widespread machine learning adaptation in all fields yet is the explainability of the model's decision-making. e.g. models far outperform human radiologists, but straight machine-learning models are not adapted in industry, due to a limited understanding of how exactly decisions are derived.
This is especially important in safety-critical applications.
@@SimonEliasen123 To add to that, even in non safety-critical applications it can be hard to work with. Because they fail in such , often weird, ways. So even when you succeed to, on average, get better results. The failures are often unpredictable and complete. Making mitigating them difficult, and much more important, because often sub optimal is much better then completely wrong.
A key limitation that should be highlighted with pure machine learning models over conceptual/theoretical models is that you can't easily interpret them and they tend to be difficult to control; meaning it's very difficult if you want to, say, see how a change in some parameters (e.g. due to climate change) might affect how the weather behaves in the future (Kaiser et al 2018). Neural nets can definitely be useful for pure operational forecasting, post processing, or for finding optimal coordinates that make your physics easy to model, interpret, and control however (Champion et al 2019).
Can't you just run neural net with this changes and see result?
Hi Simon, who would be the best people to link up with to talk about AI and weather prediction??
Awesome video and awesome topic. Maybe someone with more knowledge of climate modeling can answer me this: I would assume "classical" (i.e. non-ML) climate models also use the probabilistic method with ensembles - how many runs are there in such an ensemble?
There are always two factors in these cases: resolution and ensemble size. When you like to have a higher resolution which can give you better projections (not a must, but possible), then you have to reduce the ensemble size which gives you better opportunities to estimate the probabilities of your variables. What is the best option depends on your application. In climate models it has increased over time, from originally 3 per model in the mid 2000 to now 10s (because you also want to play with different scenarios). But there are also model simulations out there with multiple hundred ensemble members, also for the full complexity models (but usually in quite low resolution). For simple climate model you might be able to generate thousands or millions of members. The tricky question is how many ensembles should there be to estimate the probability curves alright and whether doing the simulations with different models (multi-model ensemble) offers you more insights than doing the projections with only one model. General number of ensembles nowadays: weather ~40 (some much more, some less), seasonal to decadal predictions are about 10-30 members in operational mode, climate projections are about 5-30 per model and in the order of 200 in multi-model-ensembles. Higher than that might exists in the research field, but are rarely used in operational settings (but might be one day when they proof successful).
@@markrichardson21 Thank you!
Thank you guys. That's how it looks like when YT Comments works best.
Hi Kevin, the ECMWF weather model has 51 ensemble members (50 perturbed and 1 control) and a higher resolution deterministic run
I'm a bit skeptical these will continue to pan out beyond post-processing algos, I've heard that ML models represent chaos quite poorly (as you say, it's just fancy linear regression), so until that hurdle is overcome I don't see how we can possibly get accurate weather out of this...
I think that would be more of a limit on how far into the future they can go than their other capabilities. In the future, I could see near-term high-resolution models being made with machine learning and long-range models still relying on expert knowledge.
There's also always the possibility of training convolutional auto-encoders for near-term high-resolution forecasting as a way of learning about the underlying structures to improve handcrafted models. Auto-encoders just turn out to be the best dimensional reduction technique we have and tend to be relatively understandable.
@@thamiordragonheart8682 I'm not sure it made the final cut of the video, but Andrew did mention that we tried auto-encoders and we were much less successful than we were using GANs
The PanguWeather (Huawei iirc) and GraphCast (DeepMind) papers both represent really strong steps forward in this direction
@@lucyharris2127 Cool I'll check them out! Thanks.
@@lucyharris2127 I was thinking of autoencoders more as an analysis technique for understanding processes better than as an effective stand-alone model, though it's at the edge of my data science knowledge and I think the most common autoencoder architectures aren't really meant to be used on images, which makes it harder.
I also know just enough about GANs to realize that if you're ok with a bit of a black box there isn't anything much more precise than iterating a generator and classifier against each other.
I can remember some years ago when my dear old aunt visited us in Australia. It was early September and she heard the weather forecast. "It's not likely to rain before January" Her chin nearly hit the floor, she could believe her ears. It was true that year however.
More meteorology please!!
0:01 you can't fool me with a touch of gaussian blur, i've seen the nordvpn logo too many times
Please can you do a video on Judith curry, she is one of the most outspoken scientists who downplay the affect off climate change
Just for people who want to learn more about the state of current climate models: there is a podcast (Manifold) episode called 'Status and Future of Climate Modelling' with Tim Palmer.
As a cyclist to work i hate how bad predicting rain is.
Like i can dress for the cold...
But rain is just annoying as heck.
Dont wanna sit in wet clothes all day
Does this mean you are going to add machine learning to Claude? And have there been any updates to it?
There have been big updates recently! Check out the twitch channel for a vod where I talk through the new structure. CLAUDE is a real priority for me this year, and we're not far off getting a dry dynamical core together (he says, cursing himself)
@@SimonClark Twitch? I'm old! Any chance of cross posting some of the highlights to UA-cam? I don't know what those words mean, but it sounds exciting!
@@simonabunker 👍👍👍👍👍
Wow
Does NordVPN help me see BBCPlay from outside the UK?
6:29 butterfly effect isn’t it ? 😅😝
So, what next youtube sponsorship? Raid?
that outro
I don't see how 8:42 answers your question. Is the answer that we will do?
im curious whether or not these models are open-source or not, i have no idea how weather models operate tbh. what i mean to say is that i hope its available to access for the whole world for free and doesnt prevent struggling countries from utilizing it, like the vaccine rollout
...i wont find out; i use a chrome extension called SponsorBlock...it blocks all those in show sponsors that adblock misses
Oh Simon, you're such a Nord... Nerd, I meant nerd.
Never understand how we are not utilising machine Learning to self improve the weather Forecast
People have been doing it for years: it just haven’t been obvious since weather forecasts don’t usually specify what models and other processes were involved, yet some sort of statistical/ML model will most likely have been used to generate things like spot temperatures from NWP models, usually in combination with human meteorological expertise. It’s only very recently that deep learning techniques and hardware have started to become competitive with NWP models based on actual physics, and I’m still sceptical about their ability to generalise to conditions that aren’t in their training set, but from the papers cited here it seems that they’re close to wider adoption.
The nord deal!
its unlucky than we move to black box models from white box ones...
This comment is for feeding the algorithm monster. 🍪
Seems it likes snacking on comments, replies and likes to both. 👍
Why not spread some happiness to day and share a treat or two and watch the channel grow? 💖 🌩⛈
Dam... this is fcking cool !
Linear aggression
Here is a cookie 🍪 for the algorithm. Math and computers were never good subjects for me, so I don't have anything to add.
Comment for algorithm
not sure if he is actually happy to promote NordVPN
Ofc he has a smurf army
I don't actually! That was an eavy metal marine haha, check out my painting insta "fiftyshadesofspacewolfgrey" to see my armies
It’s witchcraft I tell yeah!!
ooooooh not a fan of not explicitly stating who you are sponsored by at the start of the video
Shhhh!!! Don't tell them it's just linear regression!!!
Louis Fry Richardson served in the Friends Ambulance Unit in WWI alongside several other notable folks, including the sci-fi author Olaf Stapledon.
en.m.wikipedia.org/wiki/Friends%27_Ambulance_Unit
I really hope that during my lifetime we will still always double-check the weather forecasts with our existing models. Machine learning can make a big poopoo when you least expect it, in a way traditional phyiscs simply can't.
Are you basing your words on something specific or are you just expressing your opinion not based on facts?
@@jakubiskra523 elaborate