I'm not an AI expert by any means, I probably have made some mistakes. So I apologise in advance :) Also, I only used PyTorch to test the forward pass. Apart from that, everything else is written in pure Python (+ use of Numpy).
Hey nice stuff @Green-Code I did enjoy the video. It's nice that the UA-cam algorithm recommended it because I'm also building a nn from scratch although I'm currently debugging the hell out of it and trying to optimimze 😂, Thanks for the content amazing stuff
What does scratch mean? Does he have to make his own programming language too? Does he have to make his own computer? Does he have to design his own pcbs inside the computer? Does he have to put his own layer of silicon in a resin case and dope it? Does he have to generate his own electricity to power his house?
@@zennihilator3724 it has 2 meanings first one is from zero or step by step exp : i built my web from scratch ( means manually from zero) second is SCRATCH a web and app that help you to create simple games
3:12 the equation states that the loss of a network that returns probabilities with values from 0 to 1 is the expected output × the negative log of the actual output. The reason this works is because -log from 0 to 1 gives a higher loss as the probability approaches 0 and almost no loss as it approaches 1. Multiplying that by the expectes probability makes it so that the network only adjust the values for the outputs you want to approach 1.
It's pretty cool to implement all this from scratch. I had studied all this a few months ago but forgot most of it because I never practiced it. But this served as a refresher.
Same I had a summer class but we instead used Tensor flow and OpenCV libraries rather than coding from scratch. The concepts are very abstract but videos like these help me remember and realize what I was doing exactly.
I went through the same adventure :D I wrote a neural net from scratch in C++ just to get a deep understanding. The backpropagation part took me a while to figure out. I just got to an accuracy of 94% with MNIST, maybe because I still didn't implement optimizers and batches. Thanks for sharing :)
Having also made a neural network from scratch, I would recommend anyone else undertaking such a project to buy Neural Networks from Scratch by Kinsley and Kukiela. I don't know if this guy used that book, but its a wonderful reference.
I know nothing about this. Just a noob here in sophomore year learning maths and STEM exploring. I really really find this fascinating, although I can’t really understand it yet because the maths are beyond me right now and I suck at coding, I really want to learn this. I can just tell there’s a beauty to this I can’t yet see but I want to uncover that no matter how long it takes me.
"its getting 40-50% accuracy it sucks" I know this seems bad greencode but you just taught a computer how to recognize things that we previously thought were only recognizable by humans. thats not bad. good job.
What a great vid, new sub You summarize accurately two weeks of class of a ML Master where I didn't slept Great job doing that and understanding the fundamentals Ignore bad comments Keep the pace
loved the way how you explained all of this can you pleaseeeee make a beginner guide tutorial on how to get on to things like this exactly this type of things you got a sub
The background music makes it look like this topic is just really really cool. Or maybe it's just because I have watched gaming videos with such bg music that similar bg music videos make me feel interested in the topic.
You are the best man, you managed to turn a boring topic into a movie. I think you are going places in the content creation industry. Keep going man 🙌🥇
I just did the same thing a few days ago lol! (Also from scratch). I did it in JavaScript so I didn't even have numpy. My matrix class is like 150 lines long lol
This was super interesting! Mainly because this is something I've wanted to do myself, but I've had some troubles implementing backpropagation. Any tips on how to implement it?
*Disclaimer: After writing an explanation, I assumed you understand calculus, but all the steps are broken down and the equations are solved fully. In this video, you were introduced to the loss function. This function does as it states: Quantifying the difference or incorrectness of your Neural Network. Ideally, you want your loss function to be 0 or close to 0. Assuming your loss function is either cross-categorical cross entropy (like in this video) or a more common approach: Mean Squared Error (Which has the formula 1/2(y-o)^2 where y is your expected output and o is your actual output). *In case you want to paste this into an appropriate calculator/document and because I've been practicing; here is the LaTex version of that equation \frac{1}{2}\left(y-o ight)^{2}. Since these functions are never negative, you can assume that your loss function reaches 0 at its absolute minimum (usually you'll hit a local minimum for complicated problems). For Instance, we can tell if a function "f" hit a minimum or a maximum by checking to see if f'(x)=0 (same as 0=d/dx f(x)) or if its slope is equal to 0. Since we know that if the slope is positive at a point as x goes towards infinity then it's rising. If the slope is negative at a point as x goes towards infinity then it's falling we can find the x position of the local minimum by iteratively changing x by x->x - f '(x) * lr (lr being the learning rate). X is usually represented as the input to the function, but since we want to change the weights and not the input, we can assume that the input to your network is constant and that the weights are the input. So since you've already done the forward pass, you likely already know that a network can be structured like this: o = f (W * x). Where o is the output, f is your activation function, W is the weights matrix, and x is your vector of inputs. Assuming you're using the Mean Squared Error, let's try to find the function's derivative with respect to your weights. The entire function is 1/2(y - f (W * x))^2. The chain rule tells us that we can find the derivative dE/dW (E being the cost function) by solving dE/do * do/dW (see how do cancels). *For the sake of visuals this is the LaTex equation: \frac{d \cdot E}{d \cdot o} \cdot \frac{d \cdot o}{d \cdot W} dE/do would be represented as (y - o) or (y - f(W * x)) and do/dW would be f '(W * x) * x. Meaning that dE/dW would be (y - f(W * x)) * f '(W * x) * x. So now you can update your weights with W -> W - dE/do. And for multiple layers, you need to pass dE/dx to the next layer which in this case is (y - f(W * x)) * f '(W * x) * W (Notice how you multiply by the weights instead of the input).
I'd recommend making a class that deals with a single layer of the network. The class should have a method called backward (for backward pass or backpropagation, but you can call it whatever you want) that takes in the error and the inputs for that layer. In the method, multiply the error by the derivative of your activation function and the inputs and add that result to your weights. The method should also return the error multiplied by the derivative of your activation function BUT then multiplied by your weights (preferably before you update them). If you want, I can give you some code to reference.
@@SomethingSmellsMichy I love your explaination Understood 70% of it even with little calculus and linear algebra background as am In grade 0 but started practicing calculus and linear algebra for making my own nerual network only part I don't really get is the end, and I find it hard to code something except I can fully visualize how it works so it has been troublesome trying to fully understand it
@@UrFada I can imagine that the end starts to become more complicated as it divulges more into symbols. I was kinda trying to wrap it up because of the character limit on replies.
@@SomethingSmellsMichy Ahh yes I will try to better visualize the comment later and maybe I will able to understand but overall thank you for the explaination
I think it's funny that I'm reading this while waiting for my C++ neural network to train. My code isn't even that optimized and my program only took about 5 minutes to go through the process of training a neuron 49 billion times. It's insane how much faster C++ is at machine learning (and everything) than python.
Bro it awesome explanation and this visualization is very helpful to understand. Keep it up and I hate the maths but you skip and explain in simple it good for many people. I think you have to make video on maths for ai and machine learning need to solve
awesome video bro, congrats! I’d like to see a video on how to go from that to a generative AI or a RAG, u know? Are u planning some video like that or do u got some reference?
how did you create your character? was it written in code XD btw i like the character and simple movements, how you do that? im wanting to learn how to youtube but i'm not a fan of being in camera
Can you open source the code if you don't mind? I worked on the same project but there are some improvements I feel like I can make. Edit: I implemented SGD and got 94%-95% accuracy, with 16 hidden neurons and only a singular hidden layer (10 epochs). is it possible if you can share your model architecture? Thanks!
I have a fun fact if you have been or you know alot about computer science you would know that in the future AI is so terrifying and advance that you can't tell if you are talking to an AI or not😨
Future? Over half the total traffic on any given page on the internet is non-human bruv. We’re already there 😭 a huge portion of comments on all the social media platforms are generative AI
good video, i have a question however (might be a silly question but) would making and training a chatbot be similar to this or that would be a different thing?
I'm not an expert on that, but that has to do with transformers (which is a different neural network architecture). So it's related, but not exactly the same. If you're interested search up Andrej Karpathy. He's the goat
Ay man i used you video as a source for my report i hope you don’t mind Im in computer engineering and the report is about (Neuromorphic Computing Using FPGAs)
When you ask the network if it's a 7 or not..does it initially have a stored reference parameter set to compare with in order to know if the given photo is of a 7?
I'm not an AI expert by any means, I probably have made some mistakes. So I apologise in advance :)
Also, I only used PyTorch to test the forward pass. Apart from that, everything else is written in pure Python (+ use of Numpy).
"umm Numpy is a 'prebuilt framework'" ☝🤓
5:44 gradient descent won't ever jump like that with your implementation (without momentums), cause function in that place is rising
Is that code available in GitHub if so can you share the link plse
Building a neural network from scratch? what about IN scratch?
Hey nice stuff @Green-Code I did enjoy the video. It's nice that the UA-cam algorithm recommended it because I'm also building a nn from scratch although I'm currently debugging the hell out of it and trying to optimimze 😂,
Thanks for the content amazing stuff
Ngl using that chef hypothetical is such a neat way of explaining how a neural network functions
Yeah loved it!
lol.. "neat" way
As aways, any example or concept can be explained by using food, or anything that is food related.
Yea useful for explaining neural nets for 5 year olds...
"let's think of every neuron as a chef... Now , Let 'em cook 🗿" ahh explanation 😭
ok underrated af
now do it in scratch
What does scratch mean? Does he have to make his own programming language too? Does he have to make his own computer? Does he have to design his own pcbs inside the computer? Does he have to put his own layer of silicon in a resin case and dope it? Does he have to generate his own electricity to power his house?
Scratch.
'Website' Scratch.
Y'know... the coding language for babies...
@@zennihilator3724scratch the programming language
@@zennihilator3724that means explain all the code one by one
@@zennihilator3724
it has 2 meanings
first one is from zero or step by step
exp : i built my web from scratch ( means manually from zero)
second
is SCRATCH a web and app that help you to create simple games
I have a few different neural networks I made from scratch in excel, they run slower but really helped me understand all the math that goes into them
What how ?
I guess this is the really hard core way. I hope you're not heating your toasts on a sun :D
@@AKG58Z ah, maybe I should make a video on how to. I basically have all the formulas in there and the data set and make it run with a macro
@@joshcannon6704 yeah sure don't forget to tag me
@@joshcannon6704 link the video when u do make it
3:12 the equation states that the loss of a network that returns probabilities with values from 0 to 1 is the expected output × the negative log of the actual output. The reason this works is because -log from 0 to 1 gives a higher loss as the probability approaches 0 and almost no loss as it approaches 1. Multiplying that by the expectes probability makes it so that the network only adjust the values for the outputs you want to approach 1.
Binary Cross entropy
this is why love nerds🤩
@@Exkolix great put, same
that all went over my head bro, wish I was that smart too
It's pretty cool to implement all this from scratch. I had studied all this a few months ago but forgot most of it because I never practiced it. But this served as a refresher.
hi, where did you study it?
Same I had a summer class but we instead used Tensor flow and OpenCV libraries rather than coding from scratch.
The concepts are very abstract but videos like these help me remember and realize what I was doing exactly.
I went through the same adventure :D I wrote a neural net from scratch in C++ just to get a deep understanding. The backpropagation part took me a while to figure out. I just got to an accuracy of 94% with MNIST, maybe because I still didn't implement optimizers and batches. Thanks for sharing :)
I was really shocked in my second year when I too realised that it in fact is all math.
That’s computing in general buddy
"Wait it's all maths?"
"Always has been"
Having also made a neural network from scratch, I would recommend anyone else undertaking such a project to buy Neural Networks from Scratch by Kinsley and Kukiela. I don't know if this guy used that book, but its a wonderful reference.
I know nothing about this. Just a noob here in sophomore year learning maths and STEM exploring. I really really find this fascinating, although I can’t really understand it yet because the maths are beyond me right now and I suck at coding, I really want to learn this. I can just tell there’s a beauty to this I can’t yet see but I want to uncover that no matter how long it takes me.
"its getting 40-50% accuracy it sucks"
I know this seems bad greencode but you just taught a computer how to recognize things that we previously thought were only recognizable by humans. thats not bad. good job.
very nice mr green code very nice. You deserve a lot more subs for how good these videos are. Cant wait to see what the future holds
Honestly would love a video(series) that explains and derives the math
What a great vid, new sub
You summarize accurately two weeks of class of a ML Master where I didn't slept
Great job doing that and understanding the fundamentals
Ignore bad comments
Keep the pace
Subscribed, can't wait for such more informative videos ❤🔥
thanks! really cool! especially I just learned neural network and watching your video reinforced what I just learned from the class.
this was super cool! Keep up with these great vids! Nicely explained, convinced me to try on my own implementing a custom neural net ^^
loved the way how you explained all of this can you pleaseeeee make a beginner guide tutorial on how to get on to things like this exactly this type of things
you got a sub
+1
The background music makes it look like this topic is just really really cool. Or maybe it's just because I have watched gaming videos with such bg music that similar bg music videos make me feel interested in the topic.
ur expalination is absolutely fantastic!!
You are the best man, you managed to turn a boring topic into a movie. I think you are going places in the content creation industry. Keep going man 🙌🥇
That was cool. Now do it "in" scratch
You explained it so well, I loved it even though I understand the math behind it. ❤
It was extremely useful, thanks a lot 👍
Man mastered, If you can't explain it simply, you don't understand it well enough.
This is my first time watching you. Bro you are awesome, keep going, I love your explanation.
Hey man just wanted to find out where you made your avatar! Thanks ahead 🙏🏻
I just did the same thing a few days ago lol! (Also from scratch). I did it in JavaScript so I didn't even have numpy. My matrix class is like 150 lines long lol
Your Exaplation Is Very Simple It's Very Easy To Undertand
Dude, i'm actually in the master degree's classes, not understanding shit. I just watch your vid, and it looks bright as the sun
Mad respect for animating your every syllable! Amazing lipsyncing animation ❤❤❤
your creativity and passion shine through every project!
Now do if but where there are regions, a neuron has multiple connections and it is either on or off based on how many recent signals go into it.
Making a tutorial is one thing and adding animations is 🔥Great!!
This was super interesting! Mainly because this is something I've wanted to do myself, but I've had some troubles implementing backpropagation. Any tips on how to implement it?
*Disclaimer: After writing an explanation, I assumed you understand calculus, but all the steps are broken down and the equations are solved fully.
In this video, you were introduced to the loss function. This function does as it states: Quantifying the difference or incorrectness of your Neural Network.
Ideally, you want your loss function to be 0 or close to 0. Assuming your loss function is either cross-categorical cross entropy (like in this video) or a more common approach: Mean Squared Error (Which has the formula 1/2(y-o)^2 where y is your expected output and o is your actual output).
*In case you want to paste this into an appropriate calculator/document and because I've been practicing; here is the LaTex version of that equation \frac{1}{2}\left(y-o
ight)^{2}.
Since these functions are never negative, you can assume that your loss function reaches 0 at its absolute minimum (usually you'll hit a local minimum for complicated problems).
For Instance, we can tell if a function "f" hit a minimum or a maximum by checking to see if f'(x)=0 (same as 0=d/dx f(x)) or if its slope is equal to 0.
Since we know that if the slope is positive at a point as x goes towards infinity then it's rising. If the slope is negative at a point as x goes towards infinity then it's falling we can find the x position of the local minimum by iteratively changing x by x->x - f '(x) * lr (lr being the learning rate).
X is usually represented as the input to the function, but since we want to change the weights and not the input, we can assume that the input to your network is constant and that the weights are the input.
So since you've already done the forward pass, you likely already know that a network can be structured like this: o = f (W * x). Where o is the output, f is your activation function, W is the weights matrix, and x is your vector of inputs.
Assuming you're using the Mean Squared Error, let's try to find the function's derivative with respect to your weights. The entire function is 1/2(y - f (W * x))^2.
The chain rule tells us that we can find the derivative dE/dW (E being the cost function) by solving dE/do * do/dW (see how do cancels).
*For the sake of visuals this is the LaTex equation: \frac{d \cdot E}{d \cdot o} \cdot \frac{d \cdot o}{d \cdot W}
dE/do would be represented as (y - o) or (y - f(W * x)) and do/dW would be f '(W * x) * x. Meaning that dE/dW would be (y - f(W * x)) * f '(W * x) * x.
So now you can update your weights with W -> W - dE/do. And for multiple layers, you need to pass dE/dx to the next layer which in this case is (y - f(W * x)) * f '(W * x) * W (Notice how you multiply by the weights instead of the input).
I'd recommend making a class that deals with a single layer of the network. The class should have a method called backward (for backward pass or backpropagation, but you can call it whatever you want) that takes in the error and the inputs for that layer. In the method, multiply the error by the derivative of your activation function and the inputs and add that result to your weights. The method should also return the error multiplied by the derivative of your activation function BUT then multiplied by your weights (preferably before you update them). If you want, I can give you some code to reference.
@@SomethingSmellsMichy I love your explaination Understood 70% of it even with little calculus and linear algebra background as am In grade 0 but started practicing calculus and linear algebra for making my own nerual network only part I don't really get is the end, and I find it hard to code something except I can fully visualize how it works so it has been troublesome trying to fully understand it
@@UrFada I can imagine that the end starts to become more complicated as it divulges more into symbols. I was kinda trying to wrap it up because of the character limit on replies.
@@SomethingSmellsMichy Ahh yes I will try to better visualize the comment later and maybe I will able to understand but overall thank you for the explaination
Your code bullet is better than codebullet. You are what everyone hoped that he would be with his enigma video. Keep it up! 😂
Cleverly and simply explained, great video
Only a genius could understand how genius you are . Wow 🎉
Thank you brother. you're a LEGEND!!!
why does my acoustic pattern recognition match your avatar poses to Code Bullet?
Nice video tho, now do it in C to assert dominance
This content creator is a clone of Code Bullets style 🥱
I think it's funny that I'm reading this while waiting for my C++ neural network to train. My code isn't even that optimized and my program only took about 5 minutes to go through the process of training a neuron 49 billion times. It's insane how much faster C++ is at machine learning (and everything) than python.
@@Smurdy1 pytorch is written in C++. numpy is written in C.
@@ggsap But still, those things only make up a portion of the code in most Python AIs. The rest goes as fast as Python does.
@@Smurdy1 did you do a benchmark?
keep going mate , We believe in you
I love your explanation. Which parameter(Hyperparameter) did you tweak to move from 50% accuracy to 97% accuracy?
relu is just math.max(0,x), or just a fancy way of saying "if the output of a neuron is
Now I know why there are so many maths classes in my uni curricullam.
Bro it awesome explanation and this visualization is very helpful to understand. Keep it up and I hate the maths but you skip and explain in simple it good for many people. I think you have to make video on maths for ai and machine learning need to solve
i have a physics end sem exam tmr, this has no connection to that yet here i am watching this
You should upload more def one of new fav youtubers in general
I’ve got to say-this video is like a shot of espresso for my brain cells! ☕
9:00 the f*cking biritish moment😭😭
Cool. This explains forward back propagation. Now explain forward forward propagation.
Description: "do not click on this: "
Me: *Clicks it*
Me: *Sees a youtube page that says: "are you sure you want to leave youtube?"*
You are so wierd, yet Funny and Interesting. Loved the way u explained
the video is too gud bro , u have taken lots of efforts..
YOUR EDITING SKILLS ARE MONEY! YOU SHOULD MAKE VIDEOS IN A DIFFERNT GENRE THAN CS VIDEOS, BETTER RPM AND MORE VIEWRERS
cool just subscribed , hope u will continue to post this kind of stuff, loved your work ...
yooo, very cute video, but from who u learned all that stuff?
make it from scratch, on scratch, with a scratch, and top it with a neural network title made from SCRATCH
awesome video bro, congrats! I’d like to see a video on how to go from that to a generative AI or a RAG, u know? Are u planning some video like that or do u got some reference?
This guy has been calling us dumb for 9 minutes
Hey man I just discovered your channel and I love it and I've been wondering how long have you been do this programming I mean
so underratted channel man, keep goin!
Bruh is implementing mathematical equations like bernoullis theorems and second order differential equations 😂.
You're the best brother ❤️ 💙
I love the way this video is made.
- "Some of this guys invented the dot product"
I can't stop laughing on this one.
Bro love the video....
Can you go in-depth in the coding part. Like every step by step code you used.
Nice video, btw do you do live coding session ? Like on twitch or here on yt
Sir, u drop this sir 👑 , please wear it from next video
6:19 that is the realty of programing😭😭
I was stuck with bugs as I began training my model , until I saw this, now I'm stuck with more bugs 😅
how did you create your character? was it written in code XD
btw i like the character and simple movements, how you do that? im wanting to learn how to youtube but i'm not a fan of being in camera
I would watch a 2h video of this. I dont want cuts and entertainment.
I know, but 2h of this is a whole as movie. It takes me like 50 hours or more just to make one of these videos :)
Can you open source the code if you don't mind? I worked on the same project but there are some improvements I feel like I can make.
Edit: I implemented SGD and got 94%-95% accuracy, with 16 hidden neurons and only a singular hidden layer (10 epochs). is it possible if you can share your model architecture? Thanks!
Can you share yours? I know links in comments are impossible, but the repo name would be amazing
@@yds6268 Do you want the whole code? or do you just want an understanding/run through of what i made
I remember my first timw doing this went crazy because didnt want to do the math but wasnt so bad lol crazy good video and visualization
Your videos are really good and interesting 🔥
Excelente el video !!, muy bien explicado, brillante aporte de conocimiento. Saludos desde Rosario - Argentina.
Awesome bro. Waiting for more videos like this.
Please do more videos , i luv the way you do💥
i just realized his channel description says "Hi! I make videos about AI and programming :)" (ASCII)
Bro can u pls provide a roadmap to learn to do like you with resources, it will be a great help, thank you
College
Awesome video man. 🎉❤
Wow man, Python is love, love is Python.
I have a fun fact if you have been or you know alot about computer science you would know that in the future AI is so terrifying and advance that you can't tell if you are talking to an AI or not😨
Future? Over half the total traffic on any given page on the internet is non-human bruv. We’re already there 😭 a huge portion of comments on all the social media platforms are generative AI
Wow that's great! now do it in C.
didnt understand shit but you actually made me feel like ik a lot of shit Thanks
bro oversimplified one of the most complicated concepts.... your education is insane i hope i get to that level soon
good video, i have a question however (might be a silly question but) would making and training a chatbot be similar to this or that would be a different thing?
I'm not an expert on that, but that has to do with transformers (which is a different neural network architecture). So it's related, but not exactly the same. If you're interested search up Andrej Karpathy. He's the goat
@@Green-Code ohh okay thanks, I really am interested in learning about this ai stuff, I recently just started trying to find good ai videos
W video! Can you show off your code for the beginners here?
I really dont know wtf happend but I watched the whole video it was fun !
Amazing video bro. Nice explanation, truly 👍
Next video: Building a Neural Network Using Assembly👨💻
Backpropagation couldn't be explained simpler
Great explanation! keep on postig great stuff 😎😎
Best explained...
Someone know what the muisc at 6:35 is?
ah and also, this is seriousely such a cool video, great job mate!
Great work bro! u got another subscriber !!
Ay man i used you video as a source for my report i hope you don’t mind
Im in computer engineering and the report is about (Neuromorphic Computing Using FPGAs)
3:39 How the heck is that math, that's absolutly crazy i bet it will took people to solve this
Life of a Computer Science student 😢
When you ask the network if it's a 7 or not..does it initially have a stored reference parameter set to compare with in order to know if the given photo is of a 7?
How did you make that talking cartoon