How AI Upscaling Improves Weather Forecasts

Поділитися
Вставка
  • Опубліковано 25 гру 2024

КОМЕНТАРІ •

  • @ShaoVideoProduction
    @ShaoVideoProduction Рік тому +170

    Look mom, Simon is a real youtuber, he got the NordVPN sponsorship!

    • @xaviermaster1
      @xaviermaster1 Рік тому +14

      no a real UA-camr gets sponsored by "raid shadow legends!"

    • @davidbryden7904
      @davidbryden7904 Рік тому +5

      No, a real youtuber suffers with investment scams here in the replies! 🤪

    • @iantaakalla8180
      @iantaakalla8180 Рік тому +2

      A real UA-camr takes one video to explain their burnout

  • @carolyncopeland2722
    @carolyncopeland2722 Рік тому +30

    Given whats happening in Auckland NZ now and the past couple of days this is highly relevant and timely

    • @WojciechowskaAnna
      @WojciechowskaAnna Рік тому +8

      in case of Auckland the rainfall forecast like 1 to 2 weeks ahead would not help. There's a lot of discussion about their drain infrastructure and general urbanisation. This time nobody can't blame forecast, but more infrastructure preparation... and also crisis case preparation...

    • @dr.zoidberg8666
      @dr.zoidberg8666 Рік тому +3

      Y'all remember when billionaires were huffing on that copium? "Just move to New Zealand! Climate change can't get you if you're down there."

  • @TheReaderOnTheWall
    @TheReaderOnTheWall Рік тому +77

    I hope we'll not lose the science and the ingenuity that went behind the models, as dependent on initial conditions as they are.
    Just as generating a torrent of A.I. art can stop people from going into Art, learning techniques, lighting, composition, perspective, etc., I wish we will keep improving the principles that could let us carry on if global supply chains are not able to manifacture the insanely complicated tech needed to sustain our current system (if for example, fossile fuels came to be costlier and costlier to extract, refine, process, transport, etc.).
    Let's not surrender everything to A.I., though it's a great tool to use while we can.

    • @MeOnStuff
      @MeOnStuff Рік тому +4

      Modern physics-based weather forecasts are also run on purpose-built supercomputers.

    • @TheReaderOnTheWall
      @TheReaderOnTheWall Рік тому

      @MeOnStuff True. That's not my gripe though.

    • @benjaminkindle1841
      @benjaminkindle1841 Рік тому +3

      Personally, I don't think using AI for art and science will make art and science less popular any more than the onset of recorded music made playing instruments less popular - giving people new tools to use has the potential to compound their interest, and make the field more accessible! Another analogy could be calculators. Someone may justified in worrying that people are worse at basic arithmetic, but I believe the possibilities that the tech unlocks are well worth it.
      But of course, it's good to have some skepticism and not blindly accept all technology. Your concerns are good points and valid things to keep an eye on!

    • @Zappygunshot
      @Zappygunshot Рік тому +1

      Currently, AI is very good at taking the average out of a huge pile of data, but it cannot innovate on that by itself. Because of that, there are still many aspects of human society and existence that cannot (yet) be replaced by artificial intelligence. It can, however, make those things easier by sorting through ridiculous amounts of data in a much shorter time and without getting depressed doing it, which makes it very useful indeed, but it won't replace everything just yet. If ever.

  • @Treebark1313
    @Treebark1313 Рік тому +13

    6:35 what the network does is still a very big question mark. training the model is analogous to linear regression, but what those weights do is a much more complicated picture.

  • @MRfuzzjr101
    @MRfuzzjr101 Рік тому +14

    This is such a good video. You explained complicated ideas in very digestible terms, and the interviews were awesome! Keep it up!

  • @likebot.
    @likebot. Рік тому +3

    It's funny how I knew all my life that it doesn't rain without clouds yet never cottoned onto the idea that a raincloud can be statistically microscopic on the macroscopy of a weather forecast. I'd never understand the brilliance of this neat trick if you didn't put that out there.
    Really happy that Nord is sponsoring your video. They're Tom Scott approved, a man miserly with his endorsements.

    • @100c0c
      @100c0c Рік тому

      Why is it a neat trick?

  • @miguelguerreiro5280
    @miguelguerreiro5280 Рік тому +6

    This is very similar to what I am researching: Species distribution models. Many of the terms are the same, but instead of weather forecast, we do species forecasts.

  • @strawberrycheesecake5502
    @strawberrycheesecake5502 Рік тому +4

    It was cool to learn about a use for GAN that isn't just bypassing licensing and copyright laws. An area close to where I live got hit by desasterous rain and floods two years ago and while there where warnings, nobody knew how bad it would get. It would be great to get more accurate forecasts.

  • @DrAndrewSteele
    @DrAndrewSteele Рік тому +7

    One thing that’s easier to predict than precipitation: every machine learning researcher watching this video reeling from the sick linear regression burn

  • @kieral4098
    @kieral4098 Рік тому +3

    Love videos like this where you dive into a specific paper. Super helpful for me as I try to pick a graduate project in atmospheric and oceanic sciences!

  • @personenkenzahl
    @personenkenzahl Рік тому +4

    thanks for another brilliant video and congrats on getting the sponsorship. thumbs up.

  • @GOATMENTATOR
    @GOATMENTATOR Рік тому +4

    Dr Andrew and Dr Lucy are showing us an example of uppers and downers :D

    • @Troyseph
      @Troyseph Рік тому +1

      Hard to concentrate on one, and hard to stay interested in the other >

  • @gideonwilliams6307
    @gideonwilliams6307 Рік тому +2

    Thank you! I have loved Chaos Theory for years (brother is a math PhD, I'm an engineer) and I was hoping to see one.

  • @pokepress
    @pokepress Рік тому +1

    I’ve been working on AI-based upscaling a fair bit over the last year, so this is pretty interesting. Great to see another real-world use case.

  • @elismart13
    @elismart13 Рік тому +3

    as someone who lives in ireland this is somewhat concerning, but luckily it wasn't too bad where I lived

  • @PremierCCGuyMMXVI
    @PremierCCGuyMMXVI Рік тому +1

    I get so mad when people criticize meteorology when meteorologists and atmospheric scientists get short term weather wrong. Our atmosphere is extremely complex and it takes a ton of physics to figure it out. The good news is predictions are getting more accurate and I wish people had more confidence in science. Thank you for this, atmospheric science is extremely interesting to me.

  • @narnigrin
    @narnigrin Рік тому

    1:10 had me proper cracked up 😅 ... and with fond memories of Scotland. Where it actually snows sometimes! But not much on the coast, where I was. So ... rain, yeah.

  • @etiennelamole9565
    @etiennelamole9565 Рік тому

    5:41 Quick correction, neural network’s whole purpose is to solve NON linear problems, thus perform non linear calculations

  • @emmabird9745
    @emmabird9745 Рік тому

    Hi Simon, I've read your book and its ok. It appealed to my "who did that and how" outlook.
    In this vid you appear to be telling us that when you come up with a new model its a good idea to see what result it would have predicted on past events with past data. Good sense really.

  • @nikolai1714
    @nikolai1714 10 місяців тому

    TV's predicts what pixels will most likely be next to the pixel it has next to it, in this way it needs much less data to show a picture. That is why some times you can see lags in the colors if the colors are very similar.

  • @pokepress
    @pokepress Рік тому

    Honestly one of the places that would probably appreciate this the most is the national/state Department of Transportation. Having a better idea where precipitation will fall would massively help allocate resources more quickly and effectively.

  • @oskars.5477
    @oskars.5477 Рік тому

    I've been a fan of yours since 2017, love the content

  • @mahirdhali
    @mahirdhali Рік тому

    Sick thumbnail, man. Loved it.

  • @kyle_weather
    @kyle_weather Рік тому +1

    This meteorologist appreciates this video!

  • @Jaqaroo
    @Jaqaroo Рік тому +1

    That was great Simon! Really interesting.
    I noticed about 5 minutes into the video that there was some plinky piano background music... umm....

  • @snowboardkid202
    @snowboardkid202 Рік тому +2

    I’m glad I’m studying the PBL, I think we have a bit more work to do on our parameterizations there.. interesting video though, I hope a benefit of these “AI” models will be allowing for more hi-res nwp due to the freed up computational resources.

  • @The_budgetking
    @The_budgetking Рік тому +7

    lol congrats on the nord VPN sponsorship

  • @douglasboyle6544
    @douglasboyle6544 Рік тому +1

    It's funny when you got to the sponsor spot and the way you announced it, I was guessing it was either Raid Shadow Legends or Nord VPN

  • @toni4729
    @toni4729 Рік тому

    I can remember some years ago when my dear old aunt visited us in Australia. It was early September and she heard the weather forecast. "It's not likely to rain before January" Her chin nearly hit the floor, she could believe her ears. It was true that year however.

  • @elise3455
    @elise3455 Рік тому

    A key limitation that should be highlighted with pure machine learning models over conceptual/theoretical models is that you can't easily interpret them and they tend to be difficult to control; meaning it's very difficult if you want to, say, see how a change in some parameters (e.g. due to climate change) might affect how the weather behaves in the future (Kaiser et al 2018). Neural nets can definitely be useful for pure operational forecasting, post processing, or for finding optimal coordinates that make your physics easy to model, interpret, and control however (Champion et al 2019).

    • @jakubiskra523
      @jakubiskra523 Рік тому

      Can't you just run neural net with this changes and see result?

  • @yeetyeet7070
    @yeetyeet7070 Рік тому +2

    I really hope that during my lifetime we will still always double-check the weather forecasts with our existing models. Machine learning can make a big poopoo when you least expect it, in a way traditional phyiscs simply can't.

    • @jakubiskra523
      @jakubiskra523 Рік тому

      Are you basing your words on something specific or are you just expressing your opinion not based on facts?

    • @yeetyeet7070
      @yeetyeet7070 Рік тому +1

      @@jakubiskra523 elaborate

  • @hgriff10
    @hgriff10 Рік тому

    fabulous - the techs living in basement flats in New York City a few weeks ago would have appreciated the forecast as their flat went under water

  • @harveytheparaglidingchaser7039

    Really interesting video. Enjoyed this a lot

  • @tadhgtwo
    @tadhgtwo Рік тому

    Love the video Simon. One serious question, where did you get the jumper? Looks great.

  • @SimonEliasen123
    @SimonEliasen123 Рік тому +1

    Another reason we don't see widespread machine learning adaptation in all fields yet is the explainability of the model's decision-making. e.g. models far outperform human radiologists, but straight machine-learning models are not adapted in industry, due to a limited understanding of how exactly decisions are derived.

    • @SimonEliasen123
      @SimonEliasen123 Рік тому +1

      This is especially important in safety-critical applications.

    • @someonespotatohmm9513
      @someonespotatohmm9513 Рік тому +1

      @@SimonEliasen123 To add to that, even in non safety-critical applications it can be hard to work with. Because they fail in such , often weird, ways. So even when you succeed to, on average, get better results. The failures are often unpredictable and complete. Making mitigating them difficult, and much more important, because often sub optimal is much better then completely wrong.

  • @thesilentone4024
    @thesilentone4024 Рік тому +1

    It snowed in las vegas last night and almost every mountain around las vegas is covered in snow.

  • @ofentsewenalesego822
    @ofentsewenalesego822 Рік тому +4

    Hello Simon. This is actually quite a great video. Has there been some applications on Hydrology and water resources? And could it be used for predicting other climatic variables especially those with implications in disaster risk management?

    • @lucyharris2127
      @lucyharris2127 Рік тому +1

      I think this is a really insightful question. I have now left Oxford university, but before I left I was working on applying this technique to improve hydrological models. The other members of the group are currently working on exactly what you've mentioned here. We all think it is really important that the science is applicable to the real world.

    • @ofentsewenalesego822
      @ofentsewenalesego822 Рік тому

      @@lucyharris2127 I'd honestly love to learn and try applying it in the country that I reside. Is it possible to be guided until I'm able to navigate through. I'm early career researcher with little experience, so that would be helpful.
      Please share links to some research articles too.

    • @lucyharris2127
      @lucyharris2127 Рік тому

      @@ofentsewenalesego822 that sounds like a great project - you will have insights from living in that country to develop the model that no one else will. In terms of guidance, I'm still early in my career myself but I'm always happy to chat and help where I can!

    • @ofentsewenalesego822
      @ofentsewenalesego822 Рік тому +1

      @@lucyharris2127 great. Just sent you a connection request on another platform. We can chat from there.

  • @thecountingthot7638
    @thecountingthot7638 Рік тому +8

    I'm a bit skeptical these will continue to pan out beyond post-processing algos, I've heard that ML models represent chaos quite poorly (as you say, it's just fancy linear regression), so until that hurdle is overcome I don't see how we can possibly get accurate weather out of this...

    • @thamiordragonheart8682
      @thamiordragonheart8682 Рік тому +3

      I think that would be more of a limit on how far into the future they can go than their other capabilities. In the future, I could see near-term high-resolution models being made with machine learning and long-range models still relying on expert knowledge.
      There's also always the possibility of training convolutional auto-encoders for near-term high-resolution forecasting as a way of learning about the underlying structures to improve handcrafted models. Auto-encoders just turn out to be the best dimensional reduction technique we have and tend to be relatively understandable.

    • @lucyharris2127
      @lucyharris2127 Рік тому +1

      @@thamiordragonheart8682 I'm not sure it made the final cut of the video, but Andrew did mention that we tried auto-encoders and we were much less successful than we were using GANs

    • @lucyharris2127
      @lucyharris2127 Рік тому +1

      The PanguWeather (Huawei iirc) and GraphCast (DeepMind) papers both represent really strong steps forward in this direction

    • @thecountingthot7638
      @thecountingthot7638 Рік тому

      @@lucyharris2127 Cool I'll check them out! Thanks.

    • @thamiordragonheart8682
      @thamiordragonheart8682 Рік тому

      @@lucyharris2127 I was thinking of autoencoders more as an analysis technique for understanding processes better than as an effective stand-alone model, though it's at the edge of my data science knowledge and I think the most common autoencoder architectures aren't really meant to be used on images, which makes it harder.
      I also know just enough about GANs to realize that if you're ok with a bit of a black box there isn't anything much more precise than iterating a generator and classifier against each other.

  • @FloodSkinner
    @FloodSkinner Рік тому

    Great video, love a bit of weather.

  • @yanononopon
    @yanononopon Рік тому +1

    It's funny you're talking about super resolution as it's one of my research subjects atm. I'm doing super resolution of 3D Large Eddy Simulation data using a machine learning algorithm

  • @WAMTAT
    @WAMTAT Рік тому +1

    Amazing video

  • @ciaranmcinerney4162
    @ciaranmcinerney4162 Рік тому +1

    "It's really just linear regession" 😄
    Yessir it is...but don't tell the funder

  • @REXilef
    @REXilef Рік тому +2

    How likely is it, that this video gets uploaded the day before I start my PhD in meteorology, where a large part of my work will be with post Processing 😅

  • @_yonas
    @_yonas Рік тому

    Just for people who want to learn more about the state of current climate models: there is a podcast (Manifold) episode called 'Status and Future of Climate Modelling' with Tim Palmer.

  • @AjSmit1
    @AjSmit1 Рік тому

    0:01 you can't fool me with a touch of gaussian blur, i've seen the nordvpn logo too many times

  • @kevinpils4716
    @kevinpils4716 Рік тому +4

    Awesome video and awesome topic. Maybe someone with more knowledge of climate modeling can answer me this: I would assume "classical" (i.e. non-ML) climate models also use the probabilistic method with ensembles - how many runs are there in such an ensemble?

    • @markrichardson21
      @markrichardson21 Рік тому +7

      There are always two factors in these cases: resolution and ensemble size. When you like to have a higher resolution which can give you better projections (not a must, but possible), then you have to reduce the ensemble size which gives you better opportunities to estimate the probabilities of your variables. What is the best option depends on your application. In climate models it has increased over time, from originally 3 per model in the mid 2000 to now 10s (because you also want to play with different scenarios). But there are also model simulations out there with multiple hundred ensemble members, also for the full complexity models (but usually in quite low resolution). For simple climate model you might be able to generate thousands or millions of members. The tricky question is how many ensembles should there be to estimate the probability curves alright and whether doing the simulations with different models (multi-model ensemble) offers you more insights than doing the projections with only one model. General number of ensembles nowadays: weather ~40 (some much more, some less), seasonal to decadal predictions are about 10-30 members in operational mode, climate projections are about 5-30 per model and in the order of 200 in multi-model-ensembles. Higher than that might exists in the research field, but are rarely used in operational settings (but might be one day when they proof successful).

    • @kevinpils4716
      @kevinpils4716 Рік тому +4

      @@markrichardson21 Thank you!

    • @gehwissen3975
      @gehwissen3975 Рік тому +2

      Thank you guys. That's how it looks like when YT Comments works best.

    • @lucyharris2127
      @lucyharris2127 Рік тому +2

      Hi Kevin, the ECMWF weather model has 51 ensemble members (50 perturbed and 1 control) and a higher resolution deterministic run

  • @mtawali13
    @mtawali13 Рік тому

    Hi Simon, who would be the best people to link up with to talk about AI and weather prediction??

  • @simonabunker
    @simonabunker Рік тому +1

    Does this mean you are going to add machine learning to Claude? And have there been any updates to it?

    • @SimonClark
      @SimonClark  Рік тому +9

      There have been big updates recently! Check out the twitch channel for a vod where I talk through the new structure. CLAUDE is a real priority for me this year, and we're not far off getting a dry dynamical core together (he says, cursing himself)

    • @simonabunker
      @simonabunker Рік тому +4

      @@SimonClark Twitch? I'm old! Any chance of cross posting some of the highlights to UA-cam? I don't know what those words mean, but it sounds exciting!

    • @ShirleeKnott
      @ShirleeKnott Рік тому

      @@simonabunker 👍👍👍👍👍

  • @nikolai1714
    @nikolai1714 10 місяців тому

    I don't see how 8:42 answers your question. Is the answer that we will do?

  • @petereldergill2942
    @petereldergill2942 Рік тому

    I've been wondering why we were so delayed in being able to buy Simon's book in Canada (I got a copy at Christmas) and why the soft cover is also delayed here. What's the reasoning? I'm sure there is one I just don't understand why we had to wait for it here. Cheers from Toronto

  • @thejll
    @thejll Рік тому

    Does NordVPN help me see BBCPlay from outside the UK?

  • @powpowvideos
    @powpowvideos Рік тому

    Thank you Simon for a wonderful video! I'm strongly considering sending this video to a State Climatologist i've worked with🤣

  • @benjaminfraeyman
    @benjaminfraeyman Рік тому +1

    As a cyclist to work i hate how bad predicting rain is.
    Like i can dress for the cold...
    But rain is just annoying as heck.
    Dont wanna sit in wet clothes all day

  • @JS-pb6gb
    @JS-pb6gb Рік тому

    Please can you do a video on Judith curry, she is one of the most outspoken scientists who downplay the affect off climate change

  • @Francis_UD
    @Francis_UD Рік тому

    6:29 butterfly effect isn’t it ? 😅😝

  • @frosty_brandon
    @frosty_brandon Рік тому

    More meteorology please!!

  • @ShirleeKnott
    @ShirleeKnott Рік тому +4

    This comment is for feeding the algorithm monster. 🍪
    Seems it likes snacking on comments, replies and likes to both. 👍
    Why not spread some happiness to day and share a treat or two and watch the channel grow? 💖 🌩⛈

  • @advanceringnewholder
    @advanceringnewholder Рік тому +1

    So, what next youtube sponsorship? Raid?

  • @mauritsbol4806
    @mauritsbol4806 Рік тому +1

    that outro

  • @petersilie648
    @petersilie648 Рік тому +1

    Never understand how we are not utilising machine Learning to self improve the weather Forecast

    • @wellurban
      @wellurban Рік тому

      People have been doing it for years: it just haven’t been obvious since weather forecasts don’t usually specify what models and other processes were involved, yet some sort of statistical/ML model will most likely have been used to generate things like spot temperatures from NWP models, usually in combination with human meteorological expertise. It’s only very recently that deep learning techniques and hardware have started to become competitive with NWP models based on actual physics, and I’m still sceptical about their ability to generalise to conditions that aren’t in their training set, but from the papers cited here it seems that they’re close to wider adoption.

  • @KosmitPL
    @KosmitPL Рік тому +1

    Oh Simon, you're such a Nord... Nerd, I meant nerd.

  • @shayan_idk
    @shayan_idk Рік тому

    im curious whether or not these models are open-source or not, i have no idea how weather models operate tbh. what i mean to say is that i hope its available to access for the whole world for free and doesnt prevent struggling countries from utilizing it, like the vaccine rollout

  • @TheCosmicGuy0111
    @TheCosmicGuy0111 Рік тому +1

    Wow

  • @tamasmatyasgal5629
    @tamasmatyasgal5629 Рік тому

    its unlucky than we move to black box models from white box ones...

  • @melissamybubbles6139
    @melissamybubbles6139 Рік тому

    Here is a cookie 🍪 for the algorithm. Math and computers were never good subjects for me, so I don't have anything to add.

  • @Junoswoof
    @Junoswoof Рік тому +1

    The nord deal!

  • @賴柏勳-z9m
    @賴柏勳-z9m Місяць тому

    Hahahahaha you got the NordVPN sponsorship!

  • @ShubhamBhushanCC
    @ShubhamBhushanCC Рік тому +1

    Shhhh!!! Don't tell them it's just linear regression!!!

  • @NeoShameMan
    @NeoShameMan Рік тому

    Linear aggression

  • @sitrakaforler8696
    @sitrakaforler8696 Рік тому

    Dam... this is fcking cool !

  • @concretesmoothie9202
    @concretesmoothie9202 Рік тому

    Comment for algorithm

  • @tengkualiff
    @tengkualiff Рік тому

    Ofc he has a smurf army

    • @SimonClark
      @SimonClark  Рік тому

      I don't actually! That was an eavy metal marine haha, check out my painting insta "fiftyshadesofspacewolfgrey" to see my armies

  • @jonathandevries2828
    @jonathandevries2828 Рік тому +2

    ...i wont find out; i use a chrome extension called SponsorBlock...it blocks all those in show sponsors that adblock misses

  • @Mermaidkilla
    @Mermaidkilla Рік тому

    not sure if he is actually happy to promote NordVPN

  • @byrnemeister2008
    @byrnemeister2008 Рік тому +2

    It’s witchcraft I tell yeah!!

  • @caittastic
    @caittastic Рік тому +1

    ooooooh not a fan of not explicitly stating who you are sponsored by at the start of the video

  • @travcollier
    @travcollier Рік тому

    Louis Fry Richardson served in the Friends Ambulance Unit in WWI alongside several other notable folks, including the sci-fi author Olaf Stapledon.
    en.m.wikipedia.org/wiki/Friends%27_Ambulance_Unit

  • @KristieSaunder
    @KristieSaunder Рік тому

    I really hope that during my lifetime we will still always double-check the weather forecasts with our existing models. Machine learning can make a big poopoo when you least expect it, in a way traditional phyiscs simply can't.