When you first showed 10:32, I was thinking that ReLUs are very bad because they collapse data too much and makes points indistinguishable. However, you later showed the 3D case in 11:49, which was very insightful for me. When data lies on a low dimensional manifold of a high dimensional space, the 11:49 picture is probably more accurate. In this case, ReLUs don't actually collapse data in such a bad way.
ReLUs don't "need" to collapse data, it only collapses what is necessary. If you think in terms of linear combination of functions, 2 ReLUs can be combined into a "S" shaped sigmoid function, and 4 ReLUs can be combined to form a "Bell Curve" function. Both are very crude and merely approximations, but as you increase the number of dimensions they become smoother. An infinite amount of ReLUs (differently scaled and translated) can approximate any function.
This is actually so good! You've explained it so very clearly and left no gaps in the logic. I have been wanting to get into machine learning, and you have helped immensely.
I always tried to visualise the decision hyperplane on the data's domain, but this has been very insightful: Visualising the data into the projected-non-linear domain. Brilliant video! :)
I love these videos. All of my life I was considered mathematically stupid. I can't read mathematical notations well. I failed pre-calc. But now as an adult, watching these visual videos have led me to be able to understand those concepts that were impenetrable to me when I was younger.
This is great work from you, congratulations!! Also a big thank you to Grant Sanderson, from @3blue1brown, for manim. Both of you make quality education so much more fun, as it should be. So thanks a lot!
This is awesome!. I had half-baked knowledge on all these topics before, after watching this video it's crystal clear!. You made it look so simple. Thank You!
Really good content, one of the clearest explanations i've heard about neural networks so far! Keep up the good job, cannot wait for the following videos!
It would be nice to pay attention to the colors in the plots. I am colorblind, and at 6:52 it's almost impossible to see the boundary between the two colors.
@ゴゴ Joji Joestar ゴゴ it's not pity, it's being considerate. If you're eating some delicious food right in front of someone who can't eat, and then you start telling the person how tasty the meal is... I would say that's a little inconsiderate.
Absolutely awesome. Very informative and helpful for my visual mind. One thing I would love to see go along with your video, would be at the start you showed 3 neurons with 3 hidden layers. I would love to see a small dataset with perhaps 3 features, and follow along through with that throughout the whole video, and using real numbers so we could follow along and even work it out on paper if we wanted to. Anyways, Thank you so much for your awesome work. Subscribed!
Kids today live at the beginning of the golden age of learning. I have a vague memory of reading a quote from Einstein where he said that art and science will eventually merge to bring a new way of representing reality. If he really said that, he was right! Our visual processing capacity is vast, and if we find sophisticated visual ways of conveying complex relations (through mediums like youtube, CGI, VR, AR) we will bring about kids that have superhuman abilities of grasping reality.
@@AnishBhethanabotla I am yes, but I'm currently working on the next part! I've scripted, recorded and made most of the animations, so I have some editing and reviewing to go but it should be out soon!
Great work. I am still left wanting a more course overview which metaphorically explains how AI is not simply a series of input->hidden->output byt way of some narrative discussion and/or metaphore. Wanting that before I go deep into the linear algebra, and then reference the steps in the matrix max discussion with what was described in the initial narrative.
Great video! Can you please explain how the Scenes are added to ScreenRectangles in the intro? I checked the code on GitHub and found nothing. Is it just done using a video editing software? Thanks in advance.
Handwritten-Digits are usually grayscale images, their pixel can be written in the form rgb(a, b, c) but since it's a grayscale image, a = b = c, so I can represent each pixel with only one numerical value. Given the matrix of those values I can then shove it into a linear vector and give it to the Neural Network. My question is; given a regular image with three (or even four) values for each pixel, how do I convert that into a vector? Do I just put the values next to each other or do I need to convert the rgb values into a single integer? Well the second one seems more reasonable since the first would add additional perceptrons to the Neural Network.
One of the beauties of this idea of "learning" is that we don't actually need to worry about how we input the data, it just figures it out! Sometimes we don't even know what the data represents. As long as all the data is converted in the same way, it doesn't really matter! Here's an example: one time I used a neural network to play Super Mario Bros. The input to the network would've been a huge vector (iirc >65,000 inputs), which my laptop couldn't handle. Instead, I used the RAM of the console, which was just 128 inputs (not sure, I don't fully remember the number). However, I had no idea what each number represented! A lot of the time, our inputs are things that humans can't make sense of, but the neural network finds patterns among it. In reality, we don't use this "multilayer perceptron" I talked about in the video on colored images. Yann LeCun, the guy I talked about in the end, came up with something called "LeNet", or the convolutional neural network. I plan to make a similar video on the CNN, but it's gonna be a while before that comes out, so I encourage you to look it up! There are some fantastic resources on the web.
To answer your question, it depends. The first example of just stretching it out would be ideal, because the second one can be a lossy compression. What I mean by that is if you add them, an RGB value of (60,0,0) is the same as (0,60,0).
@@vcubingx Thanks for the reply and the great advice, I will definitely follow it. However I think you misunderstood my second "guess" because that wouldn't be a lossy compression. What I mean by that is just converting an rgb value to an integer that represents it uniquely. rgb(0, 0, 0) -> 0 rgb(255, 255, 255) -> 16777215 (256^3 - 1) Which is literally just counting every possible rgb combination. If I remember correctly there are some pretty easy binary operations to make this conversion. Would this also work? Because if so, it'd be a smaller vector. Or it ould just mess everything up hehe Anyway, I will begin to seriously study this topic, also because I have a quite strong mathematical background so it shouldn't be too hard.
Hi this explaination&visualization is awesome... Can you pls tell using which tools you created that 3d visualization coz i am desperatly looking for that to add in my Clg presentation and some of my lectures 🙂 thanks in advance
What will happen on samples on third quadrant? All of them aggregate to the central point? How this aggregation works following what we are going to lose about these samples?
11:41 - This "folding" was a bit misleading term. To me folding implies that all the distances and angles between all the points on the folded part would be preserved (rotation around the fold line). Projection would seem to be (more) accurate description of this phenomenon.
Looking at the holes between lines between the layers. I'm wondering if this is holes in the maximum logic achieved. Blur the lines you get more generalizations? Imagine light as light and not laser pointers. That is. Calculate with raytracing and scattering. Like a 3D render.
Broooo it is awesome visualisation video😍,it makes foggy image of neural network fully cristal clear , i am really waiting for next chapter bro, when you gonna upload the next chapter....
Good question! The output before the activation function is some x and y coordinate, so it's some vector [x, y]. Remember, since the output layer of the neural network has 2 outputs, we have 2 outputs between 0 and 1, essentially forming [sigma(x), sigma(y)]
since you mentioned, sigmoid and relu i was hoping you would mention the softmax activation function and probably draw the graph as well, this is good video none the less
Hi! May I ask, my 12 year old son wants to know (we are watching this together) what software are you using to make those truly wonderful animations?? Cheers from Vancouver!
Part 2 is out! ua-cam.com/video/-at7SLoVK_I/v-deo.html
We are looking for Part 3. Thx for sharing your experience and knowledge.
Did you just say you got Yann Lecun to help you!!!! He's got a TURING award boi!
Well, I think he is one of Yann's students so yeah.
Haha yeah, Yann was one of the guys who took a look at the video before I published :)
vcubingx dudeee that is so coolll....
Dang
@@vcubingx that's amazing man!
This Deep Learning Series will be a life-saver for many !
When you first showed 10:32, I was thinking that ReLUs are very bad because they collapse data too much and makes points indistinguishable. However, you later showed the 3D case in 11:49, which was very insightful for me. When data lies on a low dimensional manifold of a high dimensional space, the 11:49 picture is probably more accurate. In this case, ReLUs don't actually collapse data in such a bad way.
ReLUs don't "need" to collapse data, it only collapses what is necessary. If you think in terms of linear combination of functions, 2 ReLUs can be combined into a "S" shaped sigmoid function, and 4 ReLUs can be combined to form a "Bell Curve" function. Both are very crude and merely approximations, but as you increase the number of dimensions they become smoother. An infinite amount of ReLUs (differently scaled and translated) can approximate any function.
This is actually so good! You've explained it so very clearly and left no gaps in the logic.
I have been wanting to get into machine learning, and you have helped immensely.
I always tried to visualise the decision hyperplane on the data's domain, but this has been very insightful: Visualising the data into the projected-non-linear domain. Brilliant video! :)
Thanks!
I love these videos. All of my life I was considered mathematically stupid. I can't read mathematical notations well. I failed pre-calc. But now as an adult, watching these visual videos have led me to be able to understand those concepts that were impenetrable to me when I was younger.
This is great! Makes the analogy with biological neurons crystal clear for me for the first time 😄
Thanks! I'm happy that you understood it!
Great stuff! I'm familiar enough to understand the basics, but I love that this is visually done.
Glad you liked it!
WOW. I've been studying neural networks for a bit now, but this made me see them in a new way. PLEASE MAKE MORE VIDEOS!!!!
Really well done! It's cool to see the differences in the way you covered things compared to 3b1b. Can't wait to see more :D
Thanks a ton!
This is great work from you, congratulations!! Also a big thank you to Grant Sanderson, from @3blue1brown, for manim. Both of you make quality education so much more fun, as it should be. So thanks a lot!
Eagerly waiting for chapter 2
By far the most essential visualization of neural net Ive seen to date! 🤩
Great man, thank you so much! Can't wait to see the chapter 2!
Wow, amazing way of visualization of non linear function and how data is transformed.
Man it is just so high level. Your explanation, vizuals and the topic itself are great. Subscribed and waiting for the next chapters!!!
Thank you very much!
Neural networks look simpler than these animations made. Fantastic job!
This is awesome!. I had half-baked knowledge on all these topics before, after watching this video it's crystal clear!. You made it look so simple.
Thank You!
I liked it too. I couldn’t help notice the accordion action assiciated with the squares roots being used with two lengths to scale them.
Amazing visualization. Looking forward to next videos in the series.
Thank you!
one of the better talks on the topic. well done
Really good content, one of the clearest explanations i've heard about neural networks so far! Keep up the good job, cannot wait for the following videos!
Masterfully done, you managed to explain it in "simple terms, but not simpler"
Thank you for your video, this is one of the best videos explaining neural networks that I have seen. Good Work
Great video! You were right, even as an experienced user of neural networks it helped me see things in a different way
Glad to hear it!
One of the best and most intuitive way of descibing neural network! You took it to next level... looking forward more from you :)
Thank you!
I come from the new course by Alfredo and I dare say this was fantastic. Regards....
Fabulous! Looking forward to the next in the series.
RT from THE three blue one brown, great work dude!
Wow! I've done things with neural networks before without even understanding the actual math behind it! Very enlightening!
It would be nice to pay attention to the colors in the plots. I am colorblind, and at 6:52 it's almost impossible to see the boundary between the two colors.
Well let me tell you they were beautiful and vibrant.
@@tristunalekzander5608 ... Talk about rubbing salt to a wound..
@ゴゴ Joji Joestar ゴゴ it's not pity, it's being considerate. If you're eating some delicious food right in front of someone who can't eat, and then you start telling the person how tasty the meal is... I would say that's a little inconsiderate.
Is pretty much like the flag of Japan, with different colors. The inner circle is composed of blue dots and the remaining of red dots.
You have really developed man. Great progress and keep it up.
Absolutely awesome. Very informative and helpful for my visual mind.
One thing I would love to see go along with your video, would be at the start you showed 3 neurons with 3 hidden layers. I would love to see a small dataset with perhaps 3 features, and follow along through with that throughout the whole video, and using real numbers so we could follow along and even work it out on paper if we wanted to.
Anyways, Thank you so much for your awesome work.
Subscribed!
who came here from canziani sir course? :) Loved the visualization bro
broo this is the best explanation so far thanks
Great video. Really looking forward to the series!
Nice video.. Waiting for next chapter on visualising neural-nets !
Working on it! It should be out soon
Kids today live at the beginning of the golden age of learning. I have a vague memory of reading a quote from Einstein where he said that art and science will eventually merge to bring a new way of representing reality. If he really said that, he was right! Our visual processing capacity is vast, and if we find sophisticated visual ways of conveying complex relations (through mediums like youtube, CGI, VR, AR) we will bring about kids that have superhuman abilities of grasping reality.
Woahh, incredible! Happy to come this early :D
Another banger Vivek!
When is next part coming? Any updates?
hes in college now
@@AnishBhethanabotla I am yes, but I'm currently working on the next part! I've scripted, recorded and made most of the animations, so I have some editing and reviewing to go but it should be out soon!
This is too good. 3B1B vibes man
Superb!! Eagerly waiting for the next videos! :D
Is this Manim? Nice work!
Please make more, these are amazing!
Insane 🎉🎉,,,,more strong visualisation videos
Still waiting for the next most precious videos on the world of this series, please update.
Another excellent video🙌
OOH MY GOSHHH THIS IS GREAT CONTENT. thanks a lot!!!!
Superb animation, well done
omg, I am so glad I found this channel
Interesting, looking forward to part 2😉
Awesome video!!!
Can’t wait for the next ones!
This is amazing stuff. So many things I was able to understand that I couldn't get my head around before. Thank you so much!!! 😊
Sir, you have earned my subscribe, outstanding video.
Thanks a lot, this definetly will become handy for my study just in a bit over a month.
Great work. I am still left wanting a more course overview which metaphorically explains how AI is not simply a series of input->hidden->output byt way of some narrative discussion and/or metaphore. Wanting that before I go deep into the linear algebra, and then reference the steps in the matrix max discussion with what was described in the initial narrative.
Thanks a lot for such a vivid explanation ! Looking forward to more such content
Thank you! may i use the animation from 9:45 to 11:15 form this video for a creating a own video on this linear transformation?
@12:09 - Nice video, one question.Why didn't the sigmoid function squish the output into the unit area?
Nice! Looking forward to this series. 😁Imagine if we see an animation of an actual handwritten number image transformed into the decision "square"
Good idea! I believe distill.pub has something like this
He’s probably using publicly available data, so I doubt he’ll do an animation like that.
love your series， it‘s so great
Great video! Can you please explain how the Scenes are added to ScreenRectangles in the intro? I checked the code on GitHub and found nothing. Is it just done using a video editing software? Thanks in advance.
Awesome content man. Bravo!
awesome explanation and video, kudos!
Glad you liked it!
would be really great if this guy work out these videos with 3b1b... 3b1b is a very good educator
Handwritten-Digits are usually grayscale images, their pixel can be written in the form rgb(a, b, c) but since it's a grayscale image, a = b = c, so I can represent each pixel with only one numerical value. Given the matrix of those values I can then shove it into a linear vector and give it to the Neural Network.
My question is; given a regular image with three (or even four) values for each pixel, how do I convert that into a vector? Do I just put the values next to each other or do I need to convert the rgb values into a single integer? Well the second one seems more reasonable since the first would add additional perceptrons to the Neural Network.
One of the beauties of this idea of "learning" is that we don't actually need to worry about how we input the data, it just figures it out! Sometimes we don't even know what the data represents. As long as all the data is converted in the same way, it doesn't really matter! Here's an example: one time I used a neural network to play Super Mario Bros. The input to the network would've been a huge vector (iirc >65,000 inputs), which my laptop couldn't handle. Instead, I used the RAM of the console, which was just 128 inputs (not sure, I don't fully remember the number). However, I had no idea what each number represented! A lot of the time, our inputs are things that humans can't make sense of, but the neural network finds patterns among it.
In reality, we don't use this "multilayer perceptron" I talked about in the video on colored images. Yann LeCun, the guy I talked about in the end, came up with something called "LeNet", or the convolutional neural network. I plan to make a similar video on the CNN, but it's gonna be a while before that comes out, so I encourage you to look it up! There are some fantastic resources on the web.
To answer your question, it depends. The first example of just stretching it out would be ideal, because the second one can be a lossy compression. What I mean by that is if you add them, an RGB value of (60,0,0) is the same as (0,60,0).
@@vcubingx Thanks for the reply and the great advice, I will definitely follow it.
However I think you misunderstood my second "guess" because that wouldn't be a lossy compression.
What I mean by that is just converting an rgb value to an integer that represents it uniquely.
rgb(0, 0, 0) -> 0
rgb(255, 255, 255) -> 16777215 (256^3 - 1)
Which is literally just counting every possible rgb combination. If I remember correctly there are some pretty easy binary operations to make this conversion.
Would this also work? Because if so, it'd be a smaller vector. Or it ould just mess everything up hehe
Anyway, I will begin to seriously study this topic, also because I have a quite strong mathematical background so it shouldn't be too hard.
thank you so much for making videos on deep learning , much neded.
Simply beautiful. Thank you
Great! Looking forward to the rest
Great video, love the animations!
Hi this explaination&visualization is awesome... Can you pls tell using which tools you created that 3d visualization coz i am desperatly looking for that to add in my Clg presentation and some of my lectures 🙂 thanks in advance
Part 2 and 3 please!
Excellent. Thank you!
this is suuuuuuuuuch a great video! Thanks
Awesome job! I love it!
Thank you, UA-cam Algorithm, the Wise.
Thanks a lot
where is the next video?
Nice! Are the other 2 parts uploaded already?
Unfortunately no, I'm working to release them!
What will happen on samples on third quadrant? All of them aggregate to the central point? How this aggregation works following what we are going to lose about these samples?
At 10:30 when you apply ReLu on graph, how did you calculate which samples should stay and which one should be cut?
11:41 - This "folding" was a bit misleading term. To me folding implies that all the distances and angles between all the points on the folded part would be preserved (rotation around the fold line). Projection would seem to be (more) accurate description of this phenomenon.
Better than the 3b1b nn series
You explained it really well. Really like the videos. Plus your voice is as soothing as Sal's.
You think? Haha thank you so much, I don't think many people think that :)
When will the second video will out?
Subscribed. Excellent video!
Looking at the holes between lines between the layers. I'm wondering if this is holes in the maximum logic achieved. Blur the lines you get more generalizations? Imagine light as light and not laser pointers. That is. Calculate with raytracing and scattering. Like a 3D render.
Thanks for these videos. Visualization provides powerful ways of understanding and these videos are of great help.
Amen from the community 🙌🙌
Beautiful and intuitive!
Broooo it is awesome visualisation video😍,it makes foggy image of neural network fully cristal clear , i am really waiting for next chapter bro, when you gonna upload the next chapter....
When is the next chapter gonna come out?
Great video. When is part 2 coming out?
Very cool! But why is the output after the sigmoid function still in 2D? Doesn't each point get only one number (the probability) as output?
Good question! The output before the activation function is some x and y coordinate, so it's some vector [x, y]. Remember, since the output layer of the neural network has 2 outputs, we have 2 outputs between 0 and 1, essentially forming [sigma(x), sigma(y)]
@@vcubingx aah that makes sense! Thanks!
since you mentioned, sigmoid and relu i was hoping you would mention the softmax activation function and probably draw the graph as well, this is good video none the less
Absolutely fantastic!
Hi, Is this true: The W vector is always perpendicular to separated plane?
You just found a complex non linear boundary in my random head 🤯
Hi! May I ask, my 12 year old son wants to know (we are watching this together) what software are you using to make those truly wonderful animations?? Cheers from Vancouver!
I like this video.Thanks for sharing.
I also want to ask this same question. Which software or tool do you use to create the animations.
Nice video dude! How the hell did you make such a complex animations? :-)