What are Normalizing Flows?

Поділитися
Вставка
  • Опубліковано 15 чер 2024
  • This short tutorial covers the basics of normalizing flows, a technique used in machine learning to build up complex probability distributions by transforming simple ones.
    Timestamps:
    0:00 - Intro
    0:33 - Bijective transformation
    1:18 - Change of variables formula
    2:08 - Jacobian determinant
    4:28 - Generative model likelihood
    5:49 - Comparison with VAEs & GANs
    6:52 - NICE architecture: triangular Jacobian & coupling layers
    9:23 - Scaling matrix
    10:26 - Extensions
    Papers to check out:
    NICE: Non-linear Independent Components Estimation (arxiv.org/abs/1410.8516)
    Density estimation using Real NVP (arxiv.org/abs/1605.08803)
    Glow: Generative Flow with Invertible 1x1 Convolutions (arxiv.org/abs/1807.03039)
    Variational Inference with Normalizing Flows (arxiv.org/abs/1505.05770)
    Improving Variational Inference with Inverse Autoregressive Flow (arxiv.org/abs/1606.04934)
    Masked Autoregressive Flow for Density Estimation (arxiv.org/abs/1705.07057)
    MADE: Masked Autoencoder for Distribution Estimation (arxiv.org/abs/1502.03509)
    Discrete Flows: Invertible Generative Models of Discrete Data (arxiv.org/abs/1905.10347)
    Earlier work on flows:
    A family of non-parametric density estimation algorithms (math.nyu.edu/faculty/tabak/pu...)
    Additional reading:
    deepgenerativemodels.github.i...
    blog.evjang.com/2018/01/nf1.html
    lilianweng.github.io/lil-log/...
    akosiorek.github.io/ml/2018/04...
    Special thanks to Alex Beatson, Geoffrey Roeder, Yaniv Ovadia, Sachin Ravi, and Ryan Adams for helpful feedback on this video.
    Video style inspired by 3Blue1Brown
    Music: Trinkets by Vincent Rubinetti
    Links:
    UA-cam: / ariseffai
    Twitter: / ari_seff
    Homepage: www.ariseff.com
    If you'd like to help support the channel (completely optional), you can donate a cup of coffee via the following:
    Venmo: venmo.com/ariseff
    PayPal: www.paypal.me/ariseff
  • Наука та технологія

КОМЕНТАРІ • 85

  • @yassersouri6084
    @yassersouri6084 4 роки тому +47

    The best video on the topic I have seen so far. Well done.

  • @seank4422
    @seank4422 4 роки тому +16

    Incredible video and explanation. Felt like I was watching a 3B1B video. Thank you!

    • @tuber12321
      @tuber12321 4 роки тому +1

      Yes, it uses very similar background music!

  • @TheBlenderer
    @TheBlenderer 4 роки тому +27

    Awesome, thanks for the very clear explanation! Each step was quite "differentiable" in my head :)

  • @Terrial-tf7us
    @Terrial-tf7us 2 місяці тому

    you are amazing at explaining this concept in such a simple and understandable manner mate

  • @dbtmpl1437
    @dbtmpl1437 4 роки тому +4

    That's absolutely brilliant. Keep up the good work!

  • @michaelcarlon1831
    @michaelcarlon1831 4 роки тому +1

    This kind of video is super useful to the community! Thank you!

  • @adamconkey2771
    @adamconkey2771 3 роки тому +1

    Thank you for this nice video, I've been struggling through some blog posts and this immediately cleared some things up for me. Great work!

  • @yannickpezeu3419
    @yannickpezeu3419 3 роки тому

    Wow... I'm speechless.
    Thanks ! Amazing quality !

  • @Zokemo
    @Zokemo 4 роки тому +1

    This is really beautiful. Keep up the amazing work!

  • @poulamisinhamahapatra8104
    @poulamisinhamahapatra8104 4 роки тому

    Great visualisation of a complicated concept and lucid explanation. Thanks :)

  • @romolw6897
    @romolw6897 3 роки тому

    This is a great video! Each time I watch it I learn something new.

  • @tabesink
    @tabesink 3 роки тому

    Please put out more content! This was an amazing explanation.

  • @tiejean2851
    @tiejean2851 2 роки тому

    Thank you so much for making this video! Best video on this topic I've watched so far

  • @benren9004
    @benren9004 4 роки тому

    This is just such an elegant explanation.

  • @annasappington5911
    @annasappington5911 3 роки тому

    Fantastic video! Thanks for the hard work you put into these.

  • @prithviprakash1110
    @prithviprakash1110 3 роки тому

    Great explanation, it all makes sense now. Gonna keep come backing anytime I need to revise.

  • @matthiasherp9387
    @matthiasherp9387 2 роки тому

    Amazing explanations! I#m currently learning about normalising flows with a focus on the GLOW paper for a presentation and this video really gives a great overview und helps put different concepts together.

  • @abdjahdoiahdoai
    @abdjahdoiahdoai 9 місяців тому

    this is so good, please don’t stop making videos!

  • @ScottLeGrand
    @ScottLeGrand 4 роки тому +1

    Short, sweet, and comprehensive...

  • @davidhendriks1395
    @davidhendriks1395 2 роки тому

    Great video! Was looking for a clear explanation and this did the trick.

  • @bntejn
    @bntejn 3 роки тому

    Awesome video! Thanks for putting it together and sharing

  • @sShivam7
    @sShivam7 4 роки тому +2

    Incredible explanation!

  •  3 роки тому

    Great video! Gonna have to watch it again.

  • @user-or7ji5hv8y
    @user-or7ji5hv8y 3 роки тому

    great video! This is definitely the best video on this topic.

  • @random-anonymous
    @random-anonymous 11 місяців тому

    This is neat. Awesome graphics.. Many thanks!

  • @samuelpanzieri7867
    @samuelpanzieri7867 3 роки тому

    Great video, made a pretty difficult topic very clear!

  • @benmiss1767
    @benmiss1767 2 роки тому

    Amazing work, thank you very much!

  • @hanwei5987
    @hanwei5987 3 роки тому

    Amazing explanation & presentation :)

  • @jehillparikh
    @jehillparikh 3 роки тому

    Great video and visualisation!

  • @maximiliann.5410
    @maximiliann.5410 2 роки тому

    Thank you for the nice breakdown!

  • @kazz811
    @kazz811 3 роки тому +4

    This is some pretty high level pedagogy. Superbly done, thanks!

  • @simonguiroy6636
    @simonguiroy6636 3 роки тому

    Great video, well explained!

  • @najinajari3531
    @najinajari3531 3 роки тому

    Very clear explanation. Thanks a lot :)

  • @arrow0seb
    @arrow0seb 4 роки тому

    Great video. I hope you release more like it! :)

  • @superaluis
    @superaluis 3 роки тому

    Thanks for the great explanation!

  • @user-io1cq2gd2y
    @user-io1cq2gd2y 11 місяців тому

    the most clear I have see

  • @michaellaskin3407
    @michaellaskin3407 3 роки тому

    Such an excellent video

  • @chyldstudios
    @chyldstudios Рік тому

    Great explanation!

  • @alvinye9900
    @alvinye9900 Рік тому

    Awesome video! Thanks!

  • @huajieshao5226
    @huajieshao5226 2 роки тому

    awesome video! Like it so much!

  • @matakos22
    @matakos22 2 роки тому

    Thank you so much for this!

  • @zhenyueqin6910
    @zhenyueqin6910 4 роки тому +1

    Amazing! Thanks!

  • @eyal8615
    @eyal8615 3 роки тому

    Well explained!

  • @matthias2261
    @matthias2261 4 роки тому

    Nice! This is absolutely breakfast-appropriate.

  • @robmarks6800
    @robmarks6800 3 роки тому

    Amazing, Keep at it!

  • @jg9193
    @jg9193 3 роки тому +1

    Please make more videos like this

  • @lucasfijen
    @lucasfijen 4 роки тому +2

    Thanks a lot!

  • @brycejohnson9291
    @brycejohnson9291 3 роки тому

    that was a great video!

  • @curtisjhu
    @curtisjhu 8 місяців тому

    amazing, keep it up

  • @ChocolateMilkCultLeader
    @ChocolateMilkCultLeader 2 роки тому +1

    Please keep making videos

  • @ejkmovies594
    @ejkmovies594 3 місяці тому

    giving my 3blue1brown vibes. Amazing video.

  • @ayushgarg70
    @ayushgarg70 4 роки тому

    amazing

  • @motherbear55
    @motherbear55 3 роки тому +1

    Thanks for this explanation! Could you recommend on online class or other resource for getting a solid background in probability in order to better understand the math used to talk about generative models?

    • @sehaba9531
      @sehaba9531 6 місяців тому

      I am actually looking for the same thing, if you have found something interesting !

  • @sergicastellasape
    @sergicastellasape 4 роки тому +1

    Great explanation!! I hope more videos are coming. I have a question, I don't really understand the benefit from the coupling layer example about "partitioning the variable z into 1:d and d+1:D". As explained in the video, you still need to ensure that the lower right sub-matrix is triangular to make the jacobian fully triangular. Then, isn't just more "intuitive" to say: the transformation of each component will "only be able to look at itself and past elements"? Then any x_i will only depend on z_{1:i} so the derivative for the rest will be zero. You still need to impose this condition on the "lower right sub-jacobian", then what's the value of the initial partitioning? Thanks!

    • @ariseffai
      @ariseffai  4 роки тому +2

      Thank you and great question! The setup you describe is certainly one way of ensuring a fully triangular Jacobian and is the approach taken by autoregressive flows (e.g., arxiv.org/abs/1705.07057). But not only do we want a triangular Jacobian, we need to be able to efficiently compute its diagonal elements as well as the inverse of the overall transformation. The partitioning used by NICE is one way of yielding these two properties while still allowing for a high capacity transformation (as parameterized by m), which I think was underemphasized in the video. In the additive coupling layer, not only is the lower right sub-Jacobian triangular but it’s just the identity, giving us ones along the full diagonal. And the identity implemented by the first transformation (copying over z_{1:d} to x_{1:d}) guarantees g will be trivially invertible wrt 1st arg since the contribution from m can be recovered.

  • @ThePritt12
    @ThePritt12 3 роки тому +1

    cool video, thanks! What video editing tools do you use for the animations?

    • @ariseffai
      @ariseffai  3 роки тому

      This one used a combination of matplotlib, keynote, & FCP. I've also used manim in other videos.

  • @CristianGutierrez-th1jx
    @CristianGutierrez-th1jx Місяць тому

    Hands down the best intro to gen models one could ever had.

  • @antraxuran9
    @antraxuran9 3 роки тому

    Great video! I spotted a minor terminology mistake: you are referring to the evidence using the term "likelihood", which might confuse some folks

  • @user-qt6hr2tt9e
    @user-qt6hr2tt9e 3 роки тому

    Awesome

  • @sherlockcerebro
    @sherlockcerebro 3 роки тому

    I looked at the RealNVP and I can't seem to find the part where the latent space is smaller than the input space. Where could I find it?

  • @shiva_kondapalli
    @shiva_kondapalli 3 роки тому

    Hi! Amazing video and visualization. Curious to know if the software used for the graphics was manim?

    • @ariseffai
      @ariseffai  3 роки тому +1

      Not in this particular video, but there are several manim animations in my other videos :)

  • @junli9889
    @junli9889 3 роки тому

    @8:12 I believe here is grossed over: it seems to be the essential part, how to "make sure the lower right block is triangular"?

  • @stacksmasherninja7266
    @stacksmasherninja7266 2 роки тому

    Great video ! Can you also make a video on gaussian processes and gaussian copulas?

  • @user-or7ji5hv8y
    @user-or7ji5hv8y 3 роки тому

    How do we find such a function f that performs the transformation? Is it the neural network? If so, wouldn’t that just be a decoder?

  • @p.z.8355
    @p.z.8355 2 роки тому

    what is the connection of this to the reparametrization trick?

  • @DavidSimonTetruashvili
    @DavidSimonTetruashvili 2 роки тому

    I think there may be a typo at 5:48.
    The individual Jacobians suddenly go to be taken wrt z_i instead of x_i, in the second line. That is not so, right?

  • @saharshayegan
    @saharshayegan Рік тому

    Thank you for the great explanation. What I don't understand here is the reason why we are looking for p_theta(x). Shouldn't it be p_phi(x)? (by phi I mean any other parameter that is not theta) Since we are looking for the probability in the transformed space.

    • @ariseffai
      @ariseffai  Рік тому

      Thanks for the question. While using a single symbol for the model's parameters is a standard notation (e.g., see eq. 6 from arxiv.org/abs/1807.03039), I agree that using two distinct symbols would've been a bit clearer and indeed some papers do that instead :)

  • @user-or7ji5hv8y
    @user-or7ji5hv8y 3 роки тому

    Why would adjacent pixels for an image have autoregressive property?

  • @TyrionLannister-zz7qb
    @TyrionLannister-zz7qb Місяць тому

    Are the animations and sound track inspired from a channel named 3Blue1Brown ?

  • @karanshah1698
    @karanshah1698 2 роки тому

    Isn't the Jacobian here acting more like a Linear Transformation over the 2D example of unit square? How is it a Jacobian?
    I seem to be confused on the nomenclature here.
    Also because these are chained invertible transforms with a nonzero determinant, can't we just squash all like a Linear Transform into one?

  • @albertlee5312
    @albertlee5312 4 роки тому

    So what is normalizing flow?

  • @MDNQ-ud1ty
    @MDNQ-ud1ty 3 місяці тому +2

    I think the way you explained the probability relationships is a bit poor. For example p_t(x) = p_t(f_t^(-1)(x)) would imply the obvious desire for f_t to be the identity map. If x is a different r.v. then there is no reason one would make such a claim. The entire point is that the rv's may have different probabilities due to the map(and it may not even be injective) and so one has to scale the rv's probabilities which is where the jacobian comes in(as would a sum over the different branches).
    It would have been better to start with two different rv's and show how one could transform one in to another and the issues that might creep. E.g., This is how one would normally try to solve the problem from first principles.
    The way you set it up leaves a lot to be desired. E.g., while two rv's can easily take the same value they can have totally different probabilities which is the entire point of comparing them in this way. I don't know who would start off thinking two arbitrary rv's would have the same probabilities and sorta implying that then saying "oh wait, sike!" isn't really a good way to teach it.

  • @qichaoying4478
    @qichaoying4478 2 роки тому

    For Chinese readers, you can also refer to Doctor Li's lecture: ua-cam.com/video/uXY18nzdSsM/v-deo.html

  • @EagleHandsJonny
    @EagleHandsJonny 3 роки тому

    Got that 3blue1brown background music

  • @chadgregory9037
    @chadgregory9037 2 роки тому

    this totally has something to do with principle fibre bundles doesn't it..... this is that shit James Simons figured out back in the 70s

  • @stazizov
    @stazizov 3 місяці тому

    Hello everyone from 2024, it seems the flow-matching hype has begun

  • @wenjieyin7624
    @wenjieyin7624 3 роки тому

    one mistake: NF cannot reduce dimensions!

  • @CosmiaNebula
    @CosmiaNebula 3 роки тому +1

    The formula at 1:12 is wrong. The x on the right should be z.
    Similar for other formulas later.

    • @ariseffai
      @ariseffai  3 роки тому +1

      f is defined to be a mapping from Z to X. So f^{-1} takes x as input.

  • @gapsongg
    @gapsongg Рік тому

    Bro it is really hard to follow. Nice mic and nice video editing, but the content is way to hard. Really really hard to follow.