End To End Deep Learning Project Using MLOPS DVC Pipeline With Deployments Azure And AWS- Krish Naik
Вставка
- Опубліковано 8 лип 2024
- In this video we will be implementing an end to end Deep LEarning project using MLops tools such as DVC along with the deploymeny using giithub action in both azure and aws cloud
github: github.com/krishnaik06/Chicke...
Support me by joining membership so that I can upload these kind of videos
/ @krishnaik06
---------------------------------------------------------------------------------------------------------
Join the PWSKILLS Data Science Masters Course
Best Affordable Data Science Course From Pwskills(6-7 Months)
Impact Batch 2.0:- Data-Science-Masters (Full Stack Data Science)
1. Data Science Masters Hindi: bit.ly/3TPdrDz (Hindi)
2. Data Science Masters English: bit.ly/40gZ9hn (English)
Direct call to our team in case of any queries
+9186600 34247
+919880055539
+918147625763
+918660034247
+918951939425
▬▬▬▬▬ Contents of this video ▬▬▬▬▬
00:00:00 - Introduction
00:02:00 - Project Introduction
00:06:56 - Prerequisite
00:08:14 - Problem Statement (Chicken Disease)
00:09:52 - Project Demo
00:12:11 - Github Repository Setup
00:14:56 - Project Template Creation
00:39:43 - Requirements Installation & Project Setup
00:49:32 - Logging, Exception & Utils Modules
01:08:27 - Project Workflows
01:10:57 - Data Ingestion Notebook Experiment
01:28:10 - Data Ingestion Final Implementation
01:40:59 - Prepare Base Model Notebook Experiment
01:51:52 - Prepare Base Model Final Implementation
01:59:05 - Prepare Callbacks Notebook Experiment
02:06:27 - Prepare Callbacks Final Implementation
02:09:27 - Model Trainer Notebook Experiment
02:17:43 - Model Trainer Final Implementation
02:26:09 - Model Evaluation Notebook Experiment
02:30:00 - Model Evaluation Final Implementation
02:36:02 - Writing DVC file for tracking piplines
02:51:25 - Prediction Pipeline & User App
03:01:09 - Project CI/CD Deployment on AWS
03:31:00 - Project CI/CD Deployment on Azure
--------------------------------------------------------------------------------------------------------
►Data Science Projects:
• Now you Can Crack Any ...
►Learn In One Tutorials
Statistics in 6 hours: • Complete Statistics Fo...
Machine Learning In 6 Hours: • Complete Machine Learn...
Deep Learning 5 hours : • Deep Learning Indepth ...
►Learn In a Week Playlist
Statistics: • Live Day 1- Introducti...
Machine Learning : • Announcing 7 Days Live...
Deep Learning: • 5 Days Live Deep Learn...
NLP : • Announcing NLP Live co...
►Detailed Playlist:
Stats For Data Science In Hindi : • Starter Roadmap For Le...
Machine Learning In English : • Complete Road Map To B...
Machine Learning In Hindi : • Introduction To Machin...
Complete Deep Learning: • Why Deep Learning Is B...
No words to appreciate you and your amazing commitment.Thank you man
You are beyond amazing sir . Thank you for all the efforts .
This man's presentation good, Apart from that we are more comfortable with ur way of presentation.And Your way of teaching is very easy to understand for beginners. So please do ur own presentation video. Its so helpful for many beginners.
Whatt a video bro only live project video i have seen in my life that made me awefull great video bro it covers all the concepts and make the students strong in fundamental concepts and for newbies it takes atleast 5-10 days to just understand and execute it taies 1 month bro.
Krish, this new guy is also explaining well. All the best to him too.
Thanks for such well explained and fully implemented tutorials. It helps us freshers understand the best industry practices and crack interviews, because now, we would know what we're talking about. Please keep such tuts coming!
you are the best with the best content best implementation
and explanation
Very Nice Explanation sir... Keep going...
THE TEACHING IS AWESOME, learnt alot ... but, a game if are you are bored, drink when he says "Okay".
keep it up for other weeks too !!!!😀
You're making education engaging and accessible for everyone. #NurserytoVarsity
its been amazing for me to help out to land at first job Thanks
bro do you have an AWS Account?
Great work, thank you for uploading such type of projects.
Thank you Krish
Thank you sir for the amazing efforts. Can you please help us by creating a project with respect to Cybersecurity
Excellent video ❤
Very good, Amazing
Hey krish, loved the content!!
Is This the level-1 mlops architecture or level-0?
Your great sir
Sir i have a doubt in 1:50 th clip in the prepare_full_model method freeze_all for loop and free_till u were iterate by for layer in model.layer: u assigned "model. trainable=False" but we have to put "layer. trainable=False"
If we want to add more ML models into what changes required in Project Template and deployment strategies.... Can you extend this End to End Deployment with multiple projects ??
sir please next time use different deploy techniques like aws sagemaker and aws CodePipeline or aws lambda. we have already seen this deployment 3 or 4 times.
Hi Krish, that's an amazing tutorial. But if possible can you please do the same things with Pytorch? Please
lets try that ourself as an exercise for us.
Great video. thank you. Can you please explain what is entity and what's the use of updating entity and why we need it?
Sir, Can you provide the link for the dataset ? I think many of us are having issues in finding the dataset.
hey, where can i find the image dataset? They are not present in the github repository mentioned int he description.
Thank you very much for this amazing do you have any MLOPS and deployment implementation using LSTM for time series model?.
Amazing🎉
A video with object detection woluld be nice. Please krish please make a video
Hi Krish sir next project can you made on how to create own prompt using langchain and OpenAI
we can use this as a template for classifying other diseases?
Sir Can you also make a vedio howlow code applications build like ML Predictive analytics? can you make a Full Vedio end to end project based plz.
project explanation is great but Damn! FECAL :O
General suggestion, dont watch it while eating. 😂
Thanks for the warning bruv 😂
I should taken it serious. Now i know why you said so.
why is this?
@@oscarllerena2980 watch the video bro.
great job
sir where did you got the dataset? is the dataset available on your github repository? if then share the link sir..
please help me in my code showing cannot refs to remote main branch the folders and files are not uploaded to repository in my account on github and m using git bash windows powersheel not anaconda so which command to b typed
can such a set up scale as well?
Great
The no. of times this guy saying okay is uncountable😂😂😂😂
I am stuck at 55:30 where i am not able to import the local package into main.py....I have run "pip install -e ." already. Can anybody suggest what could be the reason?
krish his explanation is different i got used to understand ur explanation kindly will u resume
I agree the video is great, and I appreciate the author's dedication. However, a few things to improve here (sound quality and explanation) make the tutorial hard to follow.
hello , i find my self with an error at 50: 28, i am not able to install the requirements.txt, can you please give some related infromations
i am not able to download data from your repo..kindly provide google drive link or make it public
Can someone help me how to get these two data folders downloaded into system. I couldnot find them in Github. If its there can someone share the path
Shaadi Mubarakkhoo :D
29:02 I am unable to load python and then from pathlib import path. What should I do.
Hello, Thank you for the video. Do you also have similar videos of End to End deep learning project in pytorch?
After following these steps it did not work to me, furthermore I had to try to install the nvidia drivers (I could not achieve it) because torch was still working on CPU.
I, again thanks for the tutorial. I managed to get up to here, but @ 3:30:00, after accessing the public IP with 8080 port, and training (adding a "/train" and hitting enter), when I tried to classify an image, it outputs a "processing icon" (like it is thinking). At first, I though that is was doing the training or other processes before but after several, several minutes, it does not classify the input image and I do not know how to verify what the machine is doing? Any suggestions, please?
did you solve this one? 😅
@@spirit1484 Nope, I moved on onto other Krish's tutorials. But eventually I will come back to this one and I will try to solve it.
same problem here
while running the training file, i am getting thi serror
OSError: No file or directory found at artifacts\prepare_base_model\base_model_updated.h5
why i'm getting this error please help..........
After dockerfile created is it necessary run in the terminal to create docker image?
Bcz this this video not mentioned I m getting error
AttributeError: partially initialized module 'charset_normalizer' has no attribute 'md__mypyc' (most likely due to a circular import) any one tell me why this error?
Hi all, I work for a bank. How can I convert it as to suit any banking project. Any ideas....
Hi, at 01:15:00, github does not allow me to upload files bigger than 25MB. Any suggestions to work around this limitation? I saw that your dataset consists only of images of "Coccidiosis" and "Healthy" labels. Not even selecting those pictures from the whole dataset (found at Kaggle) I was able to reduce the size. Thanks in advance for any suggestion.
Google driver is not an option unless you guys manage to insert your own snipcode to download from google instead of github.
What I did is to "downsample", reduce the number of pictures from both folders up to the point to meet the 25MB limit imposed by github when uploading files. Then, I proceed with the normal coding.
Solved!
@@oscarllerena2980 Hello, I'm getting the size limit error after installing requirements.txt, it's not allowing me to commit changes into main branch. What to do?
i am getting this error....while running training file....
BoxKeyError: "'ConfigBox' object has no attribute 'training'"
please someone help me...
Where can i find the dataset . do you have link to access the dataset
?
I am trying to execute dvc file however it throws this error ERROR: failed to reproduce 'data_ingestion': output 'artifacts\data_ingestion' does not exist. Can someone tell me what should i do, i m not able find the right solution on internet.
getting this error "BoxKeyError: "'ConfigBox' object has no attribute 'artifacts_root'" when trying to execute data ingestion pipeline. Can someone tell me what should i do cuz there is no solution for this on internet.
Okay>>100000
what is the data collection method
Hi Krish and Bappy, I tried predicting this, but for healthy images also it is predicting Coccidiosis only. I tried increasing the number of epochs also. But nothing works. How to predict healthy images it correctly?
first i was facing same error, but I increased my epochs size to 3, now it is predicting healthy images also
This projects will put on resume getting job for ds fresher ???
@krishnaik06 please upload some end to end project in Krish Naik Hindi channel. Please 🙏
Okay count -> infinit
hi Krish. Thanks for the video, it is really helpful. My questions is, at 49:12, is there a special reason to use python 3.8 on the virtual environment or can it be the latest python version?
Solved!
was it necessary to use the specified version?
@@spirit1484 I think at the moment is one if not the most stable version to work with all the packages and libraries this implementation requires at the time the video was made. Either way, you can try the same implementation in other virtual environments with more updated python versions.
@@oscarllerena2980 i am getting this error....while running training file....
BoxKeyError: "'ConfigBox' object has no attribute 'training'"
please someone help me...
Used Python 3.9.6 and changing requirements for both python-box and ensure to the latest version works for me
Hi everyone,
has anyone got this error ''ERROR: failed to reproduce 'training': output 'artifacts\training\model.h5' does not exist'' at 2:47 when running command dvc repro in my case 1st two stages are running but at training stage I am getting above error.
Any response will be highly appreciated.
Thanks
What is the prerequisites to see this vedio
can someone tell how to download the dataset . iam not able to find it
why there is no link to dataset provided?
But If you terminate aws window, your link will not work, have you observe that??
yes for that u need to configure ur project on EC2 with nginx and Gunicoen,,
Hii, dvc repro is giving me an error that "No such file or directory " in stage 01 itself. any leads on it please ?
solved ! If anyone faces the issues, please search for longpath key code removal in your system.
@waism.shaikh How did you solve this issue?
krish sir.. if i follow your project line to line of code. will it work ? at the end will i able to do the same ?
Yes
you did not compile the model how you able to train the model it throughs error in my code
Faced an error "Name model is not defined" in "evaluation" function of "Evaluation" class. Use "self.model" instead of "model" to avoid error. The correct implementation is as follows:
def evaluation(self):
self.model = self.load_model(self.config.path_of_model)
self._valid_generator()
self.score = self.model.evaluate(self.valid_generator) #type: ignore
and for pytorch you have to use super init
need help sir for this error saying that ValueError: yaml file is empty.where i am doing mistake guide me
In the params.yaml type key:value
Hi Krish, I am on the half way of project can you guide me where I can download the dataset for that. Thanks, waiting for your response to continue this.
I downloaded this dataset from the kaggle but it has all images in one train folder.can you give me link of the repo to download this exact dataset
even i am trying to download the dataset, sir please provide the link to download the dataset, without dataset we cant do practice
BoxKeyError: "'ConfigBox' object has no attribute 'training'" while running training.ipynb.....please help..how can i fix it????
did you solved it?
@@faryaltahseen7197
change your python version to 3.11.4
@@faryaltahseen7197
Hi
Can I work on this project as my masters project?
I want to request Krish Naik to please make these video by themselves i have watched half of the video but still not figuring out what is going on.
GitHub repo dataset is not visible
It returns This site can't be reached
Solve this sir😢
Are there anyone who gets this error while training
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 13: invalid continuation byte
I cant loaded the updated model, i got this error
remote: Permission to entbappy/Chicken-Disease-Classification--Project.git denied to me... kindly help ,me
I can't find the data. How am i supposed to follow along without data?
Did you find the data?
@@yashpisat9267 Not yet..
Hello sir , please provide the dataset link
As thumnail says data science and video title says " deep learing "
Sir krish ur explanation is more understandable and kindly will u explain projects in upcoming videos but bappy is good too
In future it will be better pls explain with diff dl project
Bappy should get chance too to expose his talent.
What did you not like about this tutorial? I understood about everything.
@@akj3344 people are crying because it is not Krish or Sunny.
AttributeError: 'function' object has no attribute 'set_model'
why I am getting this error?
were u able to resolve that error?
i am getting this AttributeError: 'WindowsPath' object has no attribute 'endswith'
when i ruunig 03_preapre_callbacks.ipyb file. how to resolve this.
paste this in your callbacks component-
class PrepareCallback:
def __init__(self, config: PrepareCallbacksConfig):
self.config = config
@property
def _create_tb_callbacks(self):
timestamp = time.strftime("%Y-%m-%d-%H-%M-%S")
tb_running_log_dir = os.path.join(
self.config.tensorboard_root_log_dir,
f"tb_logs_at_{timestamp}",
)
return tf.keras.callbacks.TensorBoard(log_dir=tb_running_log_dir)
@property
def _create_ckpt_callbacks(self):
return tf.keras.callbacks.ModelCheckpoint(
filepath=str(self.config.checkpoint_model_filepath),
save_best_only=True,
)
def get_tb_ckpt_callbacks(self):
return [
self._create_tb_callbacks,
self._create_ckpt_callbacks
]
there's only one change in the _create_ckpt_callbacks fxn that is the filepath should be of string type i.e. filepath=str(self.config.checkpoint_model_filepath),
were u able to solve that
@@rashadptp5182 No
at 1:29:05 I'm getting error FileNotFoundError: [Errno 2] No such file or directory: 'config/config.yaml'
Can someone help in this.
make sure to be in right projects folder directory where the config.yaml file is located
Is the deployment on AWS or Azure free, someone kindly answer.
The prediction process is not working properly. both AWS and Azure.
I have reached out to all team members, but I haven't received any responses.
The deployment process has been completed successfully, and the application is running perfectly. However, the problem lies in the prediction functionality.
Please help me how to solve it...
Hey man, if you get any update do inform
did you try increasing the epochs?
i am getting issue FileNotFoundError: [Errno 2] No such file or directory: 'config\\config.yaml'
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...in 01_data_ingestion.ipynb file running as per your step
getting the same issue, somebody please help us out
Restart the notebook from the top. Than Run All. Hopefully it will resolve the issue'
Give absolute path if that works and if not please restart your kernel and choose the right env.
guys its just a simple problem. Make sure that you are in right directory
hello i am getting an error in data ingestion part because of parms.yaml being an empty file
Here is the error
ValueError Traceback (most recent call last)
Cell In[12], line 8
6 data_ingestion.extract_zip_file()
7 except Exception as e:
----> 8 raise e
Cell In[12], line 2
1 try:
----> 2 config = ConfigurationManager()
...
---> 37 raise ValueError("yaml file is empty")
38 except Exception as e:
39 raise e
ValueError: yaml file is empty
2:06:53 AttributeError: 'WindowsPath' object has no attribute 'endswith'
I too faced this issue,got rectified. You shall try this ,
In def _create_ckpt_callbacks(self):
return tf.keras.callbacks.ModelCheckpoint(
'artifacts/prepare_callbacks/checkpoint_dir/model.h5' ,
save_best_only = True
)
Were you able to fix it?
I am getting the same error.
Hey, I was able to fix it by removing the Path() from the config manager function for the callback element.
@sheikhtahamaroof8484
1 second ago
paste this in your callbacks component-
class PrepareCallback:
def __init__(self, config: PrepareCallbacksConfig):
self.config = config
@property
def _create_tb_callbacks(self):
timestamp = time.strftime("%Y-%m-%d-%H-%M-%S")
tb_running_log_dir = os.path.join(
self.config.tensorboard_root_log_dir,
f"tb_logs_at_{timestamp}",
)
return tf.keras.callbacks.TensorBoard(log_dir=tb_running_log_dir)
@property
def _create_ckpt_callbacks(self):
return tf.keras.callbacks.ModelCheckpoint(
filepath=str(self.config.checkpoint_model_filepath),
save_best_only=True,
)
def get_tb_ckpt_callbacks(self):
return [
self._create_tb_callbacks,
self._create_ckpt_callbacks
]
there's only one change in the _create_ckpt_callbacks fxn that is the filepath should be of string type i.e. filepath=str(self.config.checkpoint_model_filepath),
@@pranavmoses9923 can you explain better? Please help me
AttributeError: 'WindowsPath' object has no attribute 'endswith' why this error is coming in 03_prepare_callback.ipynb file
Hi, I am getting the same error. Did you solve the issue?
@@riachoudhari7297 Use str() around it.
Where is the given dataset?
What about ML 2023..
I am getting this error "AttributeError: 'PosixPath' object has no attribute 'endswith'" while create Callbacks, can you please help?
I'm getting the same error for 03_prepare_callbacks.ipynb file. did you find a solution for this problem?
Me too I'm facing the same problem. Anyone got a solution for it? I tried with python 3.8 and 3.10 but they're still giving me that error
Did you find a solution?
Did you find a solution? @@tarun.k5548
Did you find a solution?@@miguelhermar
please am having this error, 841 msg = (
842 "Argument {arg} of type {valt} to {f} "
...
---> 31 with open(path_to_yaml) as yaml_file:
32 content = yaml.safe_load(yaml_file)
33 logger.info(f"yaml file: {path_to_yaml} loaded successfully")
FileNotFoundError: [Errno 2] No such file or directory: 'config\\config.yaml'
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
How to solve this error
every time it says there is no module as cnnClassifier. Anyone knows why complier is throwing this error
change the imports to src.cnnClassifier, also, enter this command in your cmd/terminal: export PYTHONPATH="${PYTHONPATH}:/full/path/to/btd"
@@rhythmsaparia8788 can you explain this comand and what is full path to btd
did you found the solution?if yes please help me i have the same issue even after building the package
@krishnaik06 In git bash it is showing - bash: conda: command not found
But I have anaconda installed
Please suggest some solution
🎉👏👏