Це відео не доступне.
Перепрошуємо.
Kubeflow Pipelines - the intro!
Вставка
- Опубліковано 15 сер 2024
- Hey everyone! this is the quick introduction to kubeflow pipelines! don't worry, more are on the way! This is just a "quick" introduction!
The repo I used with all the code can be found here github.com/nul...
The kubeflow examples can be found here github.com/kub...
Thank you for watching!
![Intro to Kubeflow [kubernetes] [kubeflow]](http://i.ytimg.com/vi/cm3UHTC3qY8/mqdefault.jpg)
![Intro to Kubeflow [kubernetes] [kubeflow]](/img/tr.png)
Great video ! A video about ModelServing would be helpful, for example with KFServing. Also an End2End example (FeatureEngineering, Model creation, Serving) in Kubeflow would be very cool and helpful for own projects.
Great suggestion! I will work on that! Just a fair warning, don't expect it really soon! while I continue to work on it, when I release a full End2End solution I will want it to a) actually do something b) operate on real data and c) be 100% opensource and available. and on top of that making the video will take time itself! :) But I this is 100% on my list of things to have done. Thank you!
Great content... thanks for taking time to make these types of videos
Glad you like them! Thank you for letting me know!
Helped me a lot. Thanks!
Glad it helped! And thank you for letting me know!
kfserving with triton server, expected :) Hope someone can do this soon. And a big thanks for the videos, a lot of effort !
I have looked at triton, and do plan on making a video... eventually :) it takes quite a bit of work to learn it well enough to make a video that is more then an advert :D but maybe i'll do a first impressions/what I know now video soon?
@@NullLabs sounds like a plan ! Especially if it's considered with local storage.
great content. didn't realize it is 3 years old one, but very much useful in 2023. Thanks for your time. Can you do a video about running a spark job through Kubeflow pipeline?
Thank you! I need to get another one out! the next one is not going to be a spark job as my next one is going though "local" install options as I have been planning on doing that as of late. But i'll remember the spark job request. Kubeflow spark seems interesting!
Phew -- that was a marathon!
One thing I struggle with related to Kubeflow pipelines is the tension between keeping the containers for each pipeline step lightweight but also encouraging reuse where possible to prevent a proliferation of container images. Thoughts on striking the right balance?
I'm also planning to produce some KF related content on my channel relatively soon!
In all things balance. I generally recommend doing it just like programming, build it once in a easy way, first time you would repeat/reuse it pull it into it's own generic function/container. This has generally worked well for me.
This is a really great tutorial with a lot of personality! Could I ask (or maybe this can be more of a video request) what we can do with a pipeline once it's all set-up? I can see how it's useful for model and experiment tracking, but how do we go about exposing models as endpoints etc? Or is this not possible on the notebook server? Thanks!
Thanks, you are 100% correct that I need to do more here I have gotten busy tied up with other things. It's on my list! Thanks for asking and I just bumped it up on my list! glad it helped some!
Hi Null channel! This was super helpful, but left me thinking 'how would we do the ci/cd for kubeflow pipelines?' especially with regard prod / test separation and custom images?
This is a really good question! I'm currently working on my home setup, when it is done I do plan on making some videos around this topic. How I usually do it is like this, images get built by git repos with code, I usually build and test them with github actions but you can use whatever you like. That being said there is a gap here if you wanted to do argocd or fleet style of continuous deployment/gitops. Though we could really probably make our kubeflow pipeline be a managed resource in argocd i'm not sure if that would be horrible or not! we will have too experiment on this.
A like for the suse mug over there!
:) Thanks! now that I know ths, i'll have to fit all 3 of my SUSE mugs in each shot! ;)
Great, Thank you so much.
You are welcome! Thanks for letting me know! I will have to get some more kubeflow videos out!
A really good video ! I like your content delivery :) Thankyou
I appreciate that!
While creating a pipeline form Jupyter Notebook, when I run my notebook instance, on executing the kfp.Client.create_run_from_pipeline_func() I get the RBAC: access denied. I did delpoy the PodDefault and checked the config for 'Allow access to KF Pipelines'. Could you or anyone help. Thanks
I don't have an idea off the top of my head, do you have any more information? I would check your k8s rbac settings. :thinking: if something comes to mind I'll post it!
@@NullLabs Hey Null! Thanks for the reply. I resolved the issue and finished your tutorial. It was very informative I really appreciate your work. Basically I had to provide my pipeline host address and namespace to call create_run_from_pipeline_func()
Just curious. Which theme are you using on your current linux? Seems super cool
I use zsh + oh-my-zsh + p10k
links:
ohmyz.sh/
github.com/romkatv/powerlevel10k
Please let me know if you would like to see a video on my setup. I would say that i'm still working on it myself, but I would be more then happy to share.
p10k is a bit heavy though I really don't notice it and my cpu is kind of old and crappy, and it gives a lot of really helpful things in a nice clean way that "just works" without me fussing with things.
@@NullLabs I really love your preference. It would be super cool if you can make a video about it. Include other tools you use in your programming. Waiting curiously 😀
@@NullLabs Also Content and the way you represent it is absolutely awesome and super useful. Keep it up.
I have a notebook instance on Google cloud with some Jupiter notebooks. I want to run the notebooks remotely (for this I'm setting the instance startup script to use papermill and remotely reset the machine). Should I use kubeflow? I also want to see the progress of the notebooks and to run them one after another, remotely without going to the cloud dashboard.
I guess this depends really, kubeflow pipelines would let you do this. But there are some easier ways you might accomplish this. I say keep it simple. if all you want is the ability too run them on command you might be able to build a simple script that ssh's too the machine. I guess I would need more information on exactly what you want to do with them :) Kubeflow is a lot to just avoid loging in to google :)
can we have a link of your terminal theme ? thank's in advance
Absolutely!
For my videos I still use ZSH as it's bash comparable so everyone should be able to follow along even if they use bash.
github.com/romkatv/powerlevel10k
I use oh-my-zsh and the above theme with only minor changes to fit me. I like it.
I have started to use fish instead recently for work... I doubt I change my videos to fish, but it seems really good right now! I will probably make a video on it later this year if nothing else!
Thanks for the question and hoped this answered it!
Thank you it was helpful. can you please do a video on how to setup Kubeflow on aws ec2 ubuntu platform.
100% a solid maybe! :D
Do a video on kubeflow multi-user-isolation.
That would be a good one. I will have too think about it. It's a problem, currently kubeflow in 1.1 only fully supports it for jupyter notebooks, pipelines and the likes still don't have this issue solve. It's been a bit since my last kubeflow video. It's about time I did another one! Thanks for the suggestion!
@@NullLabs Thanks. I plan to use Kubeflow for 5 people team and currently stuck on isolating each person. I am scratching my head around this issue since a week now.
@@AjinkyaBobadepictpune So just remember, everything I show on my channel is just information and not a "one stop solution" each and every team is a little different. While you are not going to be able to separate out the pipelines, I think you should be able to give each team member their own notebooks. And I will look at making a good video on this.
@@NullLabs That would be a really good one. Also, if you could, please add a point in video where one user cannot access another users notebook.