Kolmogorov Arnold Networks (KAN) Paper Explained - An exciting new paradigm for Deep Learning?
Вставка
- Опубліковано 3 чер 2024
- This is a paper breakdown video of the paper: Kolmogorov Arnold Networks, which brilliantly provides an alternative to standard Multi Layer Perceptrons. The video discusses the main contributions and core ideas of the paper, visually explaining the math, concepts, and challenges ahead.
#deeplearning #machinelearning #neuralnetworks
To access the animations, narration scripts, slides, notes, etc for the video, consider joining us on Patreon.
Patreon: / neuralbreakdownwithavb
Timestamps:
0:00 - Intro
1:03 - Kolmogorov Arnold Representation Theorem
5:05 - KAN Layers
8:00 - Comparisons
9:00 - B-splines
11:08 - Grid Extension, Sparsification, Continual Learning
14:00 - KANs get the best of MLPs and Splines
15:00 - Advantages and Challenges for KANs
Check out the paper:
arxiv.org/abs/2404.19756
Check out code:
kindxiaoming.github.io/pykan/... and
github.com/KindXiaoming/pykan - Наука та технологія
At 7:10 there is a correction. The notations aren't consistent with the matrix shown at 5:44.
x_1 will pass through phi_{11}, phi_{21},..., phi_{51}; and x_2 would pass through phi_{21}, phi_{22},...,phi_{52}.
Basically, the activation functions should be labeled in this order: phi_{11}, phi{21}, phi{31}, phi{41}, phi{51}, phi_{21}, phi_{22}, phi_{32}, phi_{42}, phi_{52}
Credit to @bat.chev.hug.0r for pointing it out!
Simple and to-the-point explanation. You avoided the mathematical jargos cleverly.
❤
Even as a person who isn't great at math, your explanation was clear and helped me a lot in understanding this quite exciting paper! Thank you :)
2:54 The example really helps me understand...this is an amazing and simple to understand KAN. Kudos to you!
I get an itch in the back of my brain that KANs should be able to use some support-vector tricks. In particular, there should be a sub-set of training examples that support the learned splines, with the others being hit "well enough" by interpolation. It's kind of like learning the support vectors + kernel at the same time. It perhaps should be possible to train an independent KAN per minibatch with a really restricted number of free params, and use this to a) drop out the non-supporting training examples, and b) concat/combine the learned parameters recursively.
Great work bud. I also appreciate your high quality sound and gentle voice.
Excellent explanation, and great examples, thanks for sharing your knowledge !
This is an excellent explanation of the paper (now i can ease into reading the paper). Learnable activations is new and exciting and most researchers would be kicking themselves saying, "why didn't I think of that?" The next step (for the authors of the paper) may be to work with "attention", because as far as we know, that's "all you need".
Agreed! In theory, they could probably do some attention stuff when aggregating the outputs of the activation function at each layer. Instead of a regular addition, just do a (attention-)weighted addition. It'll be interesting to see for sure - Kolmogorov Arnold Attention Networks (KAAN) got a nice ring to it.
That said, I think they should prioritize making it highly parallelizable and fast first.
There have been attempts to make activation functions learnable. In my opinion, one of the most successful attempts is the radial basis function neural network. It's quite an interesting mechanism, but it is now considered outdated.
Amazing video! Great explanation & visuals. I tried to read the paper, but couldn't fully grasp it. Your video really helped my understanding.
Wow. I am sold bro. This explaination was really good.
Finally !!
A clear explanation.
Thanks bro 🇮🇶
This is the best explanation of the theorem I've found so far. I think I understood most of it when going through the paper, but this has really solidified and clarified what the proof is about.
Wonderful overview, thanks!
thank you! really good explanation and helps me a lot!
this reminds me of harmonics in sound, where the function is one-dimensional (the strength of the sound depends on time), but we can say that a sound wave is also a complex function that consists of simpler functions, namely different frequencies or harmonics of the sound wave. I have this analogy in my head
I think thats a fair analogy. I saw some stuff in Hackernews (news.ycombinator.com/item?id=40219205) where someone tried to implement a KAN layer on pytorch with Fourier coefficients (github.com/GistNoesis/FourierKAN/).
I cannot believe I actually understood this! Thank you very much ❤️👏👏👏👏🇧🇷🇧🇷🇧🇷🇧🇷
Great presentation! Impressive!
This is really well presented
Great and simple explanation. Worthy of A. Karpathy
😀
Best explanation video I've watched!
Thanks for the easy explanations
Great explanation, thanks for the video!
Best explanation I’ve seen. Thanks.
Great explanation
excellent explanation! thank you!
Awesome video! Thanks a lot 🙏
Excellent explanation!
Really clear explanation!!
It was to the point explanation
Thanks
great video! loved it.
Good explanation, please continue to make more videos on neural nets.
Thanks!
Muchas gracias por la explicación!!!
boliviano
Great video !
AVB sir you are awesome.
I loved your mathematic explanations! Thanks for this. Will sub to your patreon :)
Awesome, thank you! Glad you enjoyed it.
Do you plan on making long mathematical breakdowns and derivations of ML papers at some point in the future? An example of what I mean is something like the mathematical explanation of the diffusion model by "Outlier" UA-cam channel.
The suggestion is basically to have 2 versions of some major ML topic. An overview like this video and another one that goes into a more deep dive of derivations and simplifying it.
@@AdmMusicc Thanks for the suggestion, sounds like a good idea. I might consider doing more in-depth math videos in the future. Most of my videos right now focus on the more practical and intuitive aspects of ML algorithms with some visual cues and illustrations.
@@avb_fj Thank you!
Super explaination ❤❤
great explanation thanks. I am just a bit confuse of what contains the learnable function at the edge level and how these local parameters are updated during the backpropagation phase. Thanks !
Thank you for this great, extremely clear video. KAN network seem to be a much more sensible approach than MLP for physics as the basis function can be selected based on some prior knowledge of this field... But without GPU support it will be complicated to scale to large scale models.
very clear. Subscribed. Question though. Your b-spline visualization showed it curving under itself -- but wouldn't this make it a non continuous function? (ie/ more then 1 output per x co-ord?)
Awesome video. If possible make a video explaininb B splines in detail.😅
it's really fascinatng.
Thank you for your very clear explanation
I have a question
Please tell me about the formula at the bottom around 3:40.
In this formula, isn't "Price" the result of adding all the prices in each row above?
Or is “price” in this expression a vector?
Thanks a lot!
So in the dataset (like the Boston Housing Dataset) each row stands for one house/property with all of its different features and its price. And the task is to train a ML model that inputs the features (bedrooms/sq footage etc) and predicts the price. So basically the model can be used later when to predict prices when the price is “unknown” and the features are known. So yeah, we won’t be adding up prices from other rows because a) they are all for different houses, and b) they are the entity we wanna predict using those other attributes. Hope that answers the question!
@@avb_fj Thank you for your reply I am so glad to see
well, I think the data in the first and second rows of the Boston Housing Dataset are the input data for f1 and f2, respectively.
Then, what exactly does "Price" on the left side of this equation contain?
Or is the idea that only one line should be entered for one calculation?
this is more than good
Even though I am working on MLP in my present research work, it may be useful for my next project. nice explanation 👍😊
Good to know! All the best for your research work!
good explanation. can you show code with some kaggle data ?
How does liquid neural network compare to kan ?
Love the simplicity of explanation !
Most important that is theoreticaly prooven formula of presentation of any multidemensional function. Kolmogorov- Arnold theorem plus control points of B-spline are basis for any continuous function. Training, speed and propagation are technical problems which should be solved.
If you want to have an arbitrary complex b-spine in each node, does it mean you have an unbounded (dynamic) number of parameters?
I think the degree of the b splines is a hyperparameter that is predetermined at the start of training. So no its not dynamic during training.
That said, one can increase or decrease the number of control points of the splines after training (look at Grid Extension in the video or the paper) to create a new model with more/less complexity.
So KAR = a kind of Fourier transform for nonperiodic, multivariate functions?
It's not build of KA representation theorem but inspired from KA representation theorem
Aren't indices of the $\phi$ functions inverted at 7:10 when you describe the KAN layer? I think $x_1$ should be passed through $\phi_{11}, \phi{21},...,\phi{51}$; and the same goes for $x_2$ ? Great video anyway! Thanks
Great observation. You are correct. Thanks for pointing that out.
Is there code implementation out there yet for us plebs to play with?
Check out: kindxiaoming.github.io/pykan/intro.html and
github.com/KindXiaoming/pykan
@@avb_fj thank you so much! And, thank you for your breakdown of the math. HUGE anxiety trigger for me and your fantastic presentation skills did wonders for that.
Thanks a lot for the kind words! Fwiw, I wrestle with math and notations all the time too!
@@avb_fj You truly have a gift, thank you for sharing it with us.
how do kans work on MNIST? Because MLP can do 92% test accuracy with around 20k connections, it should be easy for you to zip mnist through it and get a result?
👍
Reminds me of wavelets. But if it doesnt work for high D datasets it's not going to be any practical use.
So basically, "promising results" on very simple function approximation. What about classifying the mnist digits? It's not even considered a meaningful test nowadays, but at least it shows that the method does work (on 28*28 dimensions). It doesn't take much time to test it.
So it is just a summation of univariate regression equations and then passing them by an activation function.
The univariate functions themselves are the trainable activation functions.
The paper also talks about MNIST. How would a CNN be represented using Kan?
For starters, they will probably try to flatten the 28x28 images into a 784 length vector and run the current version of KAN on it. Similar to how standard MLPs train on images. To do a CNN-like implementation, they will have to do more stuff like summing up representations over a rolling window/kernel, which would probably be more in the future.
What was the accuracy of Kan model on mnist tho?
@@braineaterzombie3981 they didn’t trained one for the paper… I’m sure someone online must’ve already tried it after the paper and the repo published.
Why does this sound like vector embedding (please note I've only seen small clips about vectors, so I don't know what I'm saying, just an observation)
That’s true. Any vector representation of a data point is its embedding. The outputs of a KAN layer is indeed an embedding of the input. It just computes this embedding in a different way than MLPs.
"If you want to find the secrets of the universe, think in terms of energy, frequency and vibration." Tesla
Mmh, why is it good that all functions are univariate? Why not make bivariate functions based on bezier surfaces or few-variate hyper,-surfaces? They could pack some power of combining inputs to a higher degree, while retaining some advantageous of weight changes having local impact...
I’m sure as time goes on, someone will try it to squeeze out more accuracy and performance out of KANs with multivariate functions. I guess philosophically it makes sense to keep them univariate because according to the Kolmogorov Arnold theorem, all multivariate functions is just adding up a bunch of univariate functions.
"Can KAN or can't KAN" -> like and sub 😂
its called "pascals triangle"
KANs are cool but could you please give an example of where this can be used directly ina real world scenario based on your experience?
Fffuuuuuuuu so it wasn't me the dumb one after all!!!
KAN NOT beat MLPs....they do not beat SOTA MLPs and they are too slow! MLPscan use adaptive activation functions, see the work of Jagtap et al.
I still don't understand why this paper is important. Yes, of course you can replace anything in an existing RNN style structure with some other type of function, yay. There are an innumerable number of arrangements and styles of things you could do to modify RNNs. Whee. I haven't seen a single example that shows why we should care about this particular arrangement, if KANs were good, they'd have more evidence than the very synthetic and useless examples in the paper.
So you've read and fully understood the paper?
Stop saying “less parameters”. It is “fewer parameters”.