Thanks Imran. It is so nice of you to say this and hope I was able to do video a day. I typically record during the weekend and weekdays do not find much time. Being a long weekend here I was able to record few additional videos. Will try to record as much as I can
It would be great if you could upload videos on mlflow projects and mlflow model too.As per databricks they are using it for managing complete ML lifecycle.Thanks a lot for your helpful videos again
KoolJnana.. MLFLow model is already covered where I talk about save model and load model. MLFlow projects is mostly configuration but will cover it in future
Sir, great demo and tutorial. Do you have any nice demo like this for the deployment in sagemaker and making Docker container and all you have said at the end. If so please help me in finding that and if not there, then would request you to upload them soon. will be Eagerly awaiting for the same. And what about those other components of ML Flow like MLflow project, model etc. it will be really great if you can make such tutorials for the entire ML lifecycles emphasizing the deployment part.
You can check for docker and k8s in this playlist - Machine Learning Model Deployment on Google Cloud: ua-cam.com/play/PL3N9eeOlCrP4VXtFJTjmGsqI-Emk2keVL.html Check my playlist section where you might find other videos you are looking for.
Raja.. Yes I have. Unfortunately I am not able to get rid of it completely. It comes in between when I have the code demos. Hope to get some fix soon if I find one and any suggestion welcome
Thanks for sharing an awesome video, it gonna be really helpful in the research phase. One request - can you make videos on how to production the model using kubeflow ?
Shaitender.. My experience with kubeflow is not great when creating complex ML models. Too many changes to make it to work. But will do it once I feel some of the steps in kubeflow is simplified
Thanks for making this video which I requested earlier, can you make a video on kubernetes, helm charts, kubeflow, this mlflow example is used straight from the docs, can you give a slightly advanced example for mlflow? You are doing great work thanks a lot just that my expectations have increased :)
Hi, The idea of this video is to get started directly there reading through their documentation. So yes most part of it are from docs and typically thats what most of their functionality is. I could have taken complex dataset but the video focus would have changed to model side. We can use same set of info in video and version information on data used or connect git code version. I will try to make it more detailed and integrated covering all tools together to depict entire MLOps cycle if future video. Till that time I want to provide individual tools functionality in simple way before getting to integrated way. Hope it helps?. Do let me know if any specific ask you have, I will try provided I have knowledge on it
Does MLFlow have a pipeline feature, what are the pros and cons of MLFlow vs Kubeflow vs Airflow vs Luigi vs Dask. Which tool is the best for pipelining feature which can support parallelization of the different processes, I see that Kubeflow pipeline feature is in beta and a lot of those pipeline examples out there seem to be failing due to version issues.
MLFLow is not a competitor for these but more one that complements any of your modelling task. Airflow is just an orchestrater When you say parallelization I did not get. Do you mean distributed training. Kubeflow is nothing but bunch of well know tools integrated so that one train/deploy/monitor/scale ML workloads on kubernetes. Kubeflow is long way to be real with such changing tech landscape and dependency. Also not many enterprise are Kubernetes ready. Cloud providers have their own services as well
@@AIEngineeringLife Why do you say that Kubernetes is not enterprise ready? Can you elaborate on this ? On their home page their docs are filled with case studies of companies using kubernetes. Spotify uses kubeflow but then they came up with luigi and arena also I believe. For parrallelization I meant like distributed training like Dask/Spark. What did you mean by Cloud Providers have their own services as well? Why do you think Kubeflow is long way to go? I want to use the pipelining feature which feature which tool should I look into? Dask vs Luigi vs Kubeflow for Pipelining, Orchestration will be looked by Airflow so that is settled I think.
Read my reply again :) .. I said not many enterprise are kubernetes ready did not day kube is not enterprise ready. What I meant is many are not fully invested into kubernetes yet or even if they are invested they have not gone beyond application side, data is far away
@@AIEngineeringLife A lot of startups are using Kubeflow but then I get a lot of people from the community who say to wait for 6 months. What tool would you choose over Luigi vs Dask vs Kubeflow? Which type of tasks require which tools?
suppose if client requirement is passing 6 dataframe to test their result on model in that case how we achieve that . and if we have to share this model to client as created in video how we can do that?
Hi, Most MLOps tools out there providing end to end service is not open source. I was planing to cover components of it like DVC, TFX model analysis, connect to version control, dockerizing it and kuberflow etc. Is there any framework that does complete MLOps you have in mind and is Open Source?. Many times we end up gluing many tools together and build missing component
@@AIEngineeringLife I don't know of any, I had a similar query too. I wanted to see more content on kubeflow, airflow, luigi, kubernetes, helm and fluxcd, some simple to complex examples also looking for dvc and tfx model analysis videos.
@@valerysalov8208 .. Kubeflow is nothing but bunch of components tied together running in K8s. There are lot of components getting added to kubeflow ecosystem and if you account all of it, yes it is end to end. It still lacks advanced model monitoring capability apart from Tensorboard which does not have all functionality
Have u see below playlist where I create api end point on google cloud ?. Machine Learning Model Deployment on Google Cloud: ua-cam.com/play/PL3N9eeOlCrP4VXtFJTjmGsqI-Emk2keVL.html
Thanks a lot for your tutorial. BUT First of all , you should really find a solution for the sound quality. Second , you should put english subtitles as your accent is obviously Hindi (I have no problem with that apart to understand the content) thus it's a bit "hard" to follow. If I hadn't already seen videos , I'm not sure that I will understand the content , this is the reason of my feedback. All is about a constructive criticism not for destruction. Keep It up . ALL THE BEST
Point taken and thanks for the feedback.. My initial videos had audio issue but for last 4 to 5 months I have fixed audio issue. You might not see in future as much as possible. I tried putting subtitles in initial videos and I had to do manual transcription with an external provider. It was turning out to be expensive for me As you might have noticed my channel is non advertised and so investing really is like out of my pocket. I am trying to improve my pronunciation as much as possible and will see what I can do for some of my videos I already have
Ideally I will just save a reference of data rather data itself. Data needs to have it's own version lifecycle. Say in this case once we have model I will copy the data to data_ and then put this reference in mlflow
Don't judge a "UA-cam content quality by its video length " . So much wisdom in 18 Min videos, surely gonna apply in my current project
Superb. Thanks for sharing this awesome video !
Making life easy for the Data Scientist. Thank you very much.
OMG !!! MLflow is just 18mins!!! content is really good
Very nice topic to cover, thanks much for this, learnt many things .
Very nice explanation Srinivasan. Thank you very much
Great explanation sir...thank you
very comprehensive ....great start of the day.......thanks
Thanks Rushikesh
Nowadays every day I will be waiting for ur videos
Thanks Imran. It is so nice of you to say this and hope I was able to do video a day. I typically record during the weekend and weekdays do not find much time. Being a long weekend here I was able to record few additional videos. Will try to record as much as I can
Good one
Good ML flow tutorial
Thanks a lot. Great explanation
absolutely brilliant. Super.
It would be great if you could upload videos on mlflow projects and mlflow model too.As per databricks they are using it for managing complete ML lifecycle.Thanks a lot for your helpful videos again
KoolJnana.. MLFLow model is already covered where I talk about save model and load model. MLFlow projects is mostly configuration but will cover it in future
@@AIEngineeringLife Great.Thanks a lot!!
Sir, great demo and tutorial. Do you have any nice demo like this for the deployment in sagemaker and making Docker container and all you have said at the end. If so please help me in finding that and if not there, then would request you to upload them soon. will be Eagerly awaiting for the same. And what about those other components of ML Flow like MLflow project, model etc. it will be really great if you can make such tutorials for the entire ML lifecycles emphasizing the deployment part.
You can check for docker and k8s in this playlist - Machine Learning Model Deployment on Google Cloud: ua-cam.com/play/PL3N9eeOlCrP4VXtFJTjmGsqI-Emk2keVL.html
Check my playlist section where you might find other videos you are looking for.
Excellent! Hope you observed the echo in the audio.
Raja.. Yes I have. Unfortunately I am not able to get rid of it completely. It comes in between when I have the code demos. Hope to get some fix soon if I find one and any suggestion welcome
just watched 3.33 minutes and i m writing thanks a ton :)..so much information in just 3 minutes ..what would it be in 18 :) :)
Thank you :) .. I hope the remaining 15 minutes was useful as well :)
@@AIEngineeringLife yes this is what i was looking for concise content. thanks
Thanks for sharing an awesome video, it gonna be really helpful in the research phase.
One request - can you make videos on how to production the model using kubeflow ?
Shaitender.. My experience with kubeflow is not great when creating complex ML models. Too many changes to make it to work. But will do it once I feel some of the steps in kubeflow is simplified
@@AIEngineeringLife Np , Thank you will wait :)
Thanks for making this video which I requested earlier, can you make a video on kubernetes, helm charts, kubeflow, this mlflow example is used straight from the docs, can you give a slightly advanced example for mlflow? You are doing great work thanks a lot just that my expectations have increased :)
Hi, The idea of this video is to get started directly there reading through their documentation. So yes most part of it are from docs and typically thats what most of their functionality is. I could have taken complex dataset but the video focus would have changed to model side. We can use same set of info in video and version information on data used or connect git code version. I will try to make it more detailed and integrated covering all tools together to depict entire MLOps cycle if future video. Till that time I want to provide individual tools functionality in simple way before getting to integrated way. Hope it helps?. Do let me know if any specific ask you have, I will try provided I have knowledge on it
@@AIEngineeringLife sure no issues,you can make simple videos on serving and tracking but I would like to see a more complex example not from docs
Does MLFlow have a pipeline feature, what are the pros and cons of MLFlow vs Kubeflow vs Airflow vs Luigi vs Dask. Which tool is the best for pipelining feature which can support parallelization of the different processes, I see that Kubeflow pipeline feature is in beta and a lot of those pipeline examples out there seem to be failing due to version issues.
MLFLow is not a competitor for these but more one that complements any of your modelling task. Airflow is just an orchestrater
When you say parallelization I did not get. Do you mean distributed training. Kubeflow is nothing but bunch of well know tools integrated so that one train/deploy/monitor/scale ML workloads on kubernetes. Kubeflow is long way to be real with such changing tech landscape and dependency. Also not many enterprise are Kubernetes ready. Cloud providers have their own services as well
@@AIEngineeringLife Why do you say that Kubernetes is not enterprise ready? Can you elaborate on this ? On their home page their docs are filled with case studies of companies using kubernetes. Spotify uses kubeflow but then they came up with luigi and arena also I believe.
For parrallelization I meant like distributed training like Dask/Spark.
What did you mean by Cloud Providers have their own services as well?
Why do you think Kubeflow is long way to go? I want to use the pipelining feature which feature which tool should I look into? Dask vs Luigi vs Kubeflow for Pipelining, Orchestration will be looked by Airflow so that is settled I think.
Read my reply again :) .. I said not many enterprise are kubernetes ready did not day kube is not enterprise ready. What I meant is many are not fully invested into kubernetes yet or even if they are invested they have not gone beyond application side, data is far away
@@AIEngineeringLife A lot of startups are using Kubeflow but then I get a lot of people from the community who say to wait for 6 months. What tool would you choose over Luigi vs Dask vs Kubeflow? Which type of tasks require which tools?
sir mlflow with sagemaker would also be helpful. Please make dedicated videos on that
Thanks a lot for the video.How can we run MLflow on jupyter notebook? Or Is MLflow only available on databricks?
You can do pip install mlflow in your python or conda and use it in Jupyter
@@AIEngineeringLife Thanks
Hi Sir. Nice and informative video. Will the steps be the same in the case of Google Colab?
Yes vivek steps are same. You have to install MLflow and others are same
suppose if client requirement is passing 6 dataframe to test their result on model in that case how we achieve that . and if we have to share this model to client as created in video how we can do that?
There are many mlops tools out there can you make some video which compares all mlops tools?
Hi, Most MLOps tools out there providing end to end service is not open source. I was planing to cover components of it like DVC, TFX model analysis, connect to version control, dockerizing it and kuberflow etc.
Is there any framework that does complete MLOps you have in mind and is Open Source?. Many times we end up gluing many tools together and build missing component
@@AIEngineeringLife I don't know of any, I had a similar query too. I wanted to see more content on kubeflow, airflow, luigi, kubernetes, helm and fluxcd, some simple to complex examples also looking for dvc and tfx model analysis videos.
@@AIEngineeringLife Isn't Kubeflow end to end?
@@valerysalov8208 .. Kubeflow is nothing but bunch of components tied together running in K8s. There are lot of components getting added to kubeflow ecosystem and if you account all of it, yes it is end to end. It still lacks advanced model monitoring capability apart from Tensorboard which does not have all functionality
@@AIEngineeringLife have you'll tried kubeflow?
Sir, can I load any custom pickle file from local or any other system and track it using MLFlow?
Sir please make a video how to create an api for ml model in any cloud environment
Have u see below playlist where I create api end point on google cloud ?. Machine Learning Model Deployment on Google Cloud: ua-cam.com/play/PL3N9eeOlCrP4VXtFJTjmGsqI-Emk2keVL.html
@@AIEngineeringLife Actually I was going through
You had created whole Data Playlist
Thank you sir
Thanks a lot for your tutorial.
BUT
First of all , you should really find a solution for the sound quality.
Second , you should put english subtitles as your accent is obviously Hindi (I have no problem with that apart to understand the content) thus it's a bit "hard" to follow.
If I hadn't already seen videos , I'm not sure that I will understand the content , this is the reason of my feedback.
All is about a constructive criticism not for destruction.
Keep It up .
ALL THE BEST
Point taken and thanks for the feedback.. My initial videos had audio issue but for last 4 to 5 months I have fixed audio issue. You might not see in future as much as possible. I tried putting subtitles in initial videos and I had to do manual transcription with an external provider. It was turning out to be expensive for me
As you might have noticed my channel is non advertised and so investing really is like out of my pocket. I am trying to improve my pronunciation as much as possible and will see what I can do for some of my videos I already have
@@AIEngineeringLifeGood Luck
hi , how to save particular run_id in mlflow.pyfunc.save_model
Thank you for this, can also save the data used together with the model?
Ideally I will just save a reference of data rather data itself. Data needs to have it's own version lifecycle. Say in this case once we have model I will copy the data to data_ and then put this reference in mlflow
Can you also please share the notebook?
It is in below repo Shadab
github.com/srivatsan88/UA-camLI
@@AIEngineeringLife Thanks Srivatsan :)
@@AIEngineeringLife I think you added that as html file instead of .pynb file. Please check it once
Can you please share the notebook file sir please
Here it is - github.com/srivatsan88/Mastering-Apache-Spark/blob/master/MLFlow%20Tracking%20Demo.ipynb