I did just make this one as a follow up for "running it at home" if that interests you! ua-cam.com/video/C9Cl8EcqnfE/v-deo.html and I have a kubeflow playlist with all my content around kubeflow! Thanks for the feedback!
you say this about one of my most viewed videos :D hahaha. I try not to be click baity, as well as I try to go in depth on subjects so I fear that my videos will never be very popular. I'm getting better at making videos, better at separating them up into smaller chunks (I think... though my last one had like 3 little things in it because it felt weird to make a three 4 minute videos but maybe that would have been better?) Anyhow. feel free to let me know what you don't like so I can work to make it better!
Thanks, will do! Had to take a break for a week, but looking to make my kubeflow pipelines video over the 4 day weekend I have coming up with the 4th of july holiday. If interested in kubeflow this will be one you want to watch
Oh yeah? how do you run it locally? I still have not found a way that stays up to date and is easy to maintain while letting me access it from multiple machines? To be honest I just have not figured out how to run this in a simple way to get people started on that I'm happy with.
@renanmonteirobarbosa8129 I do want a lot of things and maybe I'm asking too much. My issue when I make videos is people want to replicate and it's hard for me to have them replicate my setup.
There are quite a few, but fully opensource solutions are pretty limited. I don't know of another fully opensource project that is kubernetes native that is geared 100% too MLops. some of examples are MLFlow and Airflow. of those I like airflow but airflow is not really for machine learning and thus it has a bunch of stuff that might not be useful too you. Argo could be used on it's own but I'm not sure why that would be better for any reason unless you already manage an Argo deployment and just want to graft it onto that. Again, all of these have their trade offs, as far as I know no other solution brings the same notebook server. Some clouds have their own solution I know GCP is based off kubeflow, I am not the famillier with Azures to be honest. Sorry for the rambling, hope that answers something! Here is an artical I found that I think is good and I mostly agree with. www.datarevenue.com/en-blog/airflow-vs-luigi-vs-argo-vs-mlflow-vs-kubeflow
Quick note. I think argo and kubeflow have the largest learning curve because they require kubernetes on top of the other things... While the others don't require kubernetes but can be expanded into kubernetes
Thanks for the answer! What about a managed solution like Sagemaker from AWS? Also, I deployed Kubeflow on my local machine, but found that it was kind clumsy to control resource usage, most of the pods didn't have request limits. Also, a lot of stuff is deployed along. That left me with a feeling that's better to have a production cluster destinated for Kubeflow. Am I wrong? What's your suggestion on a production environment? (eks, k8s, docker, company using multiple k8s environments)
Awesome questions. In (very) short. I do think you should have clusters just for kubeflow... But I generally think you should be running multiple clusters anyhow. I still consider kubeflow to early access because of some of the issues you brought up, though there are quick ways to fix some of it.
@@NullLabs Do you know by any change how to get an output of a run using the KFP SDK via code (i.e. not on the Google Cloud platform) Been struggling for ages with this. Thank you in advance
@@drakZes This is actually a really good question. To be honest I don't have a good idea off the top of my head. I know a few ways I might do it but they all seem... rough... Let me think about it, I'll make a video if I think of anything good or if you check out my latest videos there is a link to discord you can join and I can let you know sooner then getting a completed video out the door!
great stuff! thanks for doing this, please continue with kf
Thanks! Will do. Hopefully I'll get at least one KF out a week for a few weeks.
great overview dude!
Thanks!
Thanks for the overview with pros and cons.
I did just make this one as a follow up for "running it at home" if that interests you! ua-cam.com/video/C9Cl8EcqnfE/v-deo.html and I have a kubeflow playlist with all my content around kubeflow! Thanks for the feedback!
maybe a very important video with not much views.
you say this about one of my most viewed videos :D hahaha. I try not to be click baity, as well as I try to go in depth on subjects so I fear that my videos will never be very popular. I'm getting better at making videos, better at separating them up into smaller chunks (I think... though my last one had like 3 little things in it because it felt weird to make a three 4 minute videos but maybe that would have been better?) Anyhow. feel free to let me know what you don't like so I can work to make it better!
I’ll do the local install, that sounds cool
I'll try to have that out as soon as I can! Currently working on some documentation updates to make this even better on the kubeflow website!
Great videos - keep it up!
Thanks, will do! Had to take a break for a week, but looking to make my kubeflow pipelines video over the 4 day weekend I have coming up with the 4th of july holiday. If interested in kubeflow this will be one you want to watch
its always worth it to run kubeflow.
Oh yeah? how do you run it locally? I still have not found a way that stays up to date and is easy to maintain while letting me access it from multiple machines? To be honest I just have not figured out how to run this in a simple way to get people started on that I'm happy with.
@@NullLabs you wanna to many things. That’s why.
In the open source world and considering nvidia software. Kubeflow is the best to get started.
@renanmonteirobarbosa8129 I do want a lot of things and maybe I'm asking too much. My issue when I make videos is people want to replicate and it's hard for me to have them replicate my setup.
What are the alternatives to Kubeflow?
There are quite a few, but fully opensource solutions are pretty limited. I don't know of another fully opensource project that is kubernetes native that is geared 100% too MLops. some of examples are MLFlow and Airflow. of those I like airflow but airflow is not really for machine learning and thus it has a bunch of stuff that might not be useful too you. Argo could be used on it's own but I'm not sure why that would be better for any reason unless you already manage an Argo deployment and just want to graft it onto that. Again, all of these have their trade offs, as far as I know no other solution brings the same notebook server. Some clouds have their own solution I know GCP is based off kubeflow, I am not the famillier with Azures to be honest. Sorry for the rambling, hope that answers something! Here is an artical I found that I think is good and I mostly agree with. www.datarevenue.com/en-blog/airflow-vs-luigi-vs-argo-vs-mlflow-vs-kubeflow
Quick note. I think argo and kubeflow have the largest learning curve because they require kubernetes on top of the other things... While the others don't require kubernetes but can be expanded into kubernetes
Thanks for the answer! What about a managed solution like Sagemaker from AWS? Also, I deployed Kubeflow on my local machine, but found that it was kind clumsy to control resource usage, most of the pods didn't have request limits. Also, a lot of stuff is deployed along. That left me with a feeling that's better to have a production cluster destinated for Kubeflow. Am I wrong? What's your suggestion on a production environment? (eks, k8s, docker, company using multiple k8s environments)
This one post from you is probably enough for 10 hours of videos from me.
Awesome questions. In (very) short. I do think you should have clusters just for kubeflow... But I generally think you should be running multiple clusters anyhow. I still consider kubeflow to early access because of some of the issues you brought up, though there are quick ways to fix some of it.
Two bleeding edges sounds messy
yeah. Hopefully I did not make it sound too bad. just wanted people to know what they where getting into! :)
picturing a large pool of blood in my case.
need kubeflow pipelines videos
I will work on them soon. It's on my list to do in the next few weeks. Thanks for letting me know!
@@NullLabs appreciate the work :)
I like your comments on why we shouldn't go to Kubernetes in certain situations.. Very relatable
So
awesome
@@NullLabs Do you know by any change how to get an output of a run using the KFP SDK via code (i.e. not on the Google Cloud platform) Been struggling for ages with this. Thank you in advance
@@drakZes This is actually a really good question. To be honest I don't have a good idea off the top of my head. I know a few ways I might do it but they all seem... rough... Let me think about it, I'll make a video if I think of anything good or if you check out my latest videos there is a link to discord you can join and I can let you know sooner then getting a completed video out the door!