How to Schedule & Automatically Run Python Code!
Вставка
- Опубліковано 7 чер 2024
- The first 1000 people to use the link will get a free trial of Skillshare Premium Membership: skl.sh/keithgalli11201
In this video we learn various methods for how we can schedule and automatically run python code. We'll start by looking at cronjobs & windows task scheduler and then we will walk through running tasks in the cloud using AWS Lambda & Cloudwatch. The ultimate goal of this video is to learn how we can automatically generate & send an email with an analytics report every day.
There are methods highlighted in this video that will work on Mac, Linux, & Windows.
~ GitHub resources ~
Link to source code: github.com/KeithGalli/schedul...
Link to analytics report code: github.com/KeithGalli/generat...
~ Relevant videos ~
How to generate an analytics report in python: • How to Generate an Ana...
How to send an email with python (@coreyms ): • How to Send Emails Usi...
~ Articles mentioned ~
AWS Lambda memory limits: hackernoon.com/exploring-the-...
Downloading python linux libraries on Windows: / lambda-with-pandas
~ Other resources ~
Python email library docs: docs.python.org/3.8/library/e...
AWS Cron info (scheduling): docs.aws.amazon.com/AmazonClo...
Lambda package setup: docs.aws.amazon.com/lambda/la...
Thank you to Skillshare for sponsoring this video!
-------------------------
Video timeline!
0:00 - What we’ll be doing in this video
0:56 - Check out Skillshare! (sponsored)
1:56 - How can we automate scripts? Overview of local, cloud, and serverless methods
5:18 - Simple example of local script automation w/ cronjobs & windows task scheduler
18:32 - How to schedule code on a cloud machine (use cronjobs)
18:51 - Simple example of cloud script automation w/ AWS Lambda & Cloudwatch
27:09 - Schedule & automate sending an email locally
45:12 - Schedule & automate sending an email on the cloud w/ Lambda & Cloudwatch
50:18 - Installing python packages in a serverless environment (zip uploads)
55:50 - Generate & schedule sending analytics reports (locally)
1:02:45 - Generate & schedule sending analytics reports in AWS Lambda
1:07:03 - Limitations of lambda (max file upload size)
1:09:00 - Generate & schedule sending analytics reports in AWS Lambda
1:18:32 - Final thoughts & video recap!
-------------------------
Follow me on social media!
Instagram | / keithgalli
Twitter | / keithgalli
-------------------------
If you are curious to learn how I make my tutorials, check out this video: • How to Make a High Qua...
Practice your Python Pandas data science skills with problems on StrataScratch!
stratascratch.com/?via=keith
Join the Python Army to get access to perks!
UA-cam - / @keithgalli
Patreon - / keithgalli
*I use affiliate links on the products that I recommend. I may earn a purchase commission or a referral bonus from the usage of these links.
Hope everyone is doing well! If you have any questions about the material covered, let me know in the comments :)
Also, be sure to check out Skillshare: skl.sh/keithgalli11201. The first 1000 people to use the link will get a free trial of Skillshare Premium Membership!
Hello Keith Galli,
Could you record a quick video explaining how to schedule a script running on the jupyter notebook? You will help me a lot...
Thanks, I appreciate your work here on UA-cam.
Hey could u make a video on food classification and recommendation combined? Classify a meal and recommend recipe plzzz
I hope you do an advanced course in Keras with custom models and custom compile
I just found your channel a few days ago and it’s already become my favorite python channel. I really love the real world examples. Fantastic content. Please keep it up.
Keith continues to read my mind with video ideas..
Lol great minds think alike :)
Exactly
This is an amazing resource. I'm using what I am learning from these videos for projects in my internship. Thank you!!!
You are so committed in what you do in all your videos. Thank you so much since I wanted a script I wrote that taps into SQL server -> dB -> Tables -> data frames and dashboard output to run automated....great timing and awesome video
I LITERALLY just needed this solution. Let’s go Keith
Man, I just want to say thank you. You showed me python, now i'm looking for job in Data Science and i'm much above my friends from studys on that specialization, so...
THANK YOU!
Thank you for your clear delivery and explanations, loved it. Gave me what i was looking for plus also taught me about other methods and just more general knowledge about python windows and linux. I wish you all the success and happiness you desire
This is incredibly useful
great job as always
thank you
i really appreciate the creative and useful tutorial topics
This is great. Better than paid tutorials out there. Thanks for providing it for free.
Amazing Content. I appreciate the effort!!
Well done! Bro, Keep making such beautiful Videos
Awesome vids from this channel. Keep it up!
Great stuff as always Keith! I have learned more from you than during my whole engineering education at the university!
Awesome 👍 thanks for making this so simple.🤘
This is very nice, thank you!
Thanks for these great videos
I'm new into web-scraping. For school, I'm trying to create a project comparing food prices in in different settings like Chick-Fil-A in stores vs Chick-Fil-A in airports/malls. I watched your other web-scraping videos but I think a video about scraping then converting it over to an excel/csv file would be very nice! Thank you for all the content you do post!!!
Thank you for your knowledge good Sir.
This video showed up on my recommendations at the right time. UA-cam reading my mind. I've been trying to wrap my head around Cloud Functions for a while but had not seen any suitable examples until now. Only problem for me, though, is that I want to be able to use Google Cloud Functions, and it's not really clicking for me to translate what you show with AWS Lambda to GCP Cloud Functions. I'm hoping it'll "click" at some point since I'd rather use the GCP Ecosystem. If it doesn't, I'll switch over to AWS Lambda since what you show in this video does seem to make good sense and will make for a good template for my particular projects. Thanks!
That's a really good video thank you
Sick video ...really fun one...just a suggestion could you upload videos more often..also i know u are busy with your job
thank you !
I hope some day you will do a video on custom models in Keras and other advanced stuff
Awesome! thanks
Keith , how to motivate an absolute beginner to start delving into this giant ocean of programming, honestly feel overwhelmed just opening a book or watching a video, even your amazing videos. THANKS
Just do it
Hello Cambridge, I would like to give you an advice.
Say you want to exercise but there are so many different sports that you don't know where to start... well not doing anything is not a good start, see what fits you better or what you like and then try-out. Indeed programming is a "giant ocean", so first you should know what area/s interests you. Select a broad area (for example web development or data science) and then go deeper or even change/combine paths as you advance.
Motivation is second tier, what you need is being constant and resilient. You don't build a house on a motivated day, each day you place some bricks and after sometime, you have a house. Sure, some days you may be more motivated so you advance faster, but being constant means that even in a day you are not motivated at-all you still make progress and that is key for getting closer to your goal. Being resilient helps since (I assume you don't come from a programming background, neither did I) some concepts might get complicated so a strong mindset is needed to continue learning and not quitting or waiting again till you feel motivated.
@@danielrv1325 THANK YOU KINDLY DANIEL! REALLY GREAT ADVICE. THANKS AGAIN.
@@cambridgebreaths3581 wow where’s my thank you? Let me just say that, even though I spent a lot of time with sites like freecodecamp and sololearn, I didn’t really become what I would consider to be a semi competent programmer until I had to do it for my job. I was inheriting a lot of python scripts from a boss who left and that’s when I realized I could use it to make stuff that was really cool and practical. I used the scripts he built to get a basic and practical foundation, and then I went to Keith’s videos to expand my skills.
There’s just all sorts of cool shit you can do with python that is practically useful. For instance, I used my basic knowledge to pull covid data from an api and analyze it, which is something I used to do with Excel but that was so slow and lame. I can also use api requests to make tables of marijuana strains and their genetics, which is something I’m interested in. There are so many practical guides on UA-cam to build a project from start to finish: just copying one of those line by line will build your skills and all you have to do is modify a few things to make wholly different projects.
It’s not about just gaining knowledge for its own sake but making practically useful things. You’re lucky to be diving into this during a time when a lot of the tutorials are ready made and there is a huge knowledge base on sites like stack exchange that will answer most questions you have. So just do it. Build something
@@vestborobaptistchurch3161 Thanks Vestbord. This one programming advice, to just do it and start programming, (overlooked inadvertently!), summaries the best programming advices.Thanks again
WHOAH HERE BEFORE 100k GEETH
Thanks for another practical walkthrough, Keith. Question here-I'm on a windows and trying to set up task scheduler with a python script. Running into an "no module named selenium" (example as thats one of the modules in my script) but I had set both python and anaconda to my path. Also tried a batch file where I first list the python.exe path along with my script but that also doesnt work. Any obvious pointers? Thanks in advance man
Hey Keith, would it be possible to cover making a python program into a service? I have some monitoring tasks that I'd like to run that continue to monitor files in a folder. It would be really nice to make this into service / background task.
Man I love your vids you are the best! Its hard to get a nice answer because everyone calls themselves professionals these days but you are the gold. So here goes my question: Is there any real great book/books or courses you can suggest for python oop + algorithms in general? Also a personal question >> which programming languages you think I should crash? In school we started with python but I find c++ so fun as well.. or what other languages I should master for getting a good job and being well skilled? Really hoping you answer me.Thank you 😭🥺
Hi, can you make a video about typical questions, which a data scientist has to answer?
Hey Keith! Amazing video man...totally love it! I was trying out the Dog photo messaging which you just showed in it and I was getting only the url of the photo gotten from the api instead of the photo when the message pops up in my phone. Can you help me with it!? Thanks
Nice video. I leave a Pi 4 running 24/7 for CRON jobs and web scraping.
Ps why did you use port 465 for smtp and not 587 ? I thought 465 was deprecated.
Awesome your 🙏
Hey Keith! You should do a hair tutorial! Get a bowl cut or mullet!
We don't need to install python specially for Kali app, if we already have it for windows, right?
Hi, is any playlist like Data Science or Machine learning in 100 days. Thanks
I could use that to set up links to pages to share my UA-cam videos
Hello Keith, I have followed your instructions to schedule a run script from phyton but Im unable to find out what is wrong... the scheduler simply does not run the script... i have tried with .exe files and do but not the .py.... running the .py outside the shell performs as expected.
What would you recommend?
Thank you.
Thomas
great
@KeithGalli thanks for the video but i am having a problem I am having python file and i did the same crontab as shown in video i am not ale to run the file automatically i can see cron job it is proper but now i am not able to understand what went wrong [not from coding background]
Hi Keith please could you please show how to run in azure pipeline ? 😢
I'm here because I want to do a similar task to texting yourself a picture of a dog!
Hi Keith! I keep having an error message on AWS Lambda like this 'SMTPAuthenticationError'. It looks like I can't login into my gmail, but it works if I run the code in the terminal. I also enabled access to less secure apps in gmail, but still it doesn't work. I was wondering if you have had this problem before or not.
Cute dog!
😊
Hi Keith quick question. To schedule a Jupyter notebook job do I need to save it as a python py format first and then use the window scheduler? or can I do it on the notebook directly using the import schedule function? If I use the import schedule function say I run it on virtual machine do I need to keep it open in my browser in order to work? Thank Keith
Were you able to find a solution for this?
19:40 Best way to host and run phython script for free with AWS
Hey Keith. I am running into an inssue using the scheduling method via AWS. I have python code that flips through 3 phone numbers, but when I schedule it to run every day at 11am my code is executed from the starting phone number. And instead of having 1 Log stream with each event every day at 11am a new logstream is created everyday. Why might this be happening?
I use a raspberry PI zero to run my scripts and cronjob. Hardly any overhead and I can leave it 24/7!
Great combo!
Whenever I make a automated python script I keep running into Windows User Account Control popup. Python can't do ANYTHING from here. How do I use python to work around this so I can make a useful python script?
Surprised this didn't include the Python module "schedule!"
Keith I want to schedule my python code every week at a particular time . Request you to please suggest what to do
I recommend hosting a cheap machine in the cloud and running a cron job at a weekly interval. I think this is the simplest to setup, reliable solution
Hey Keith i need help my Sublime text 3 always errors it always shows this: Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
[Finished in 0.5s] If any of the viewers know this you can help fix it
Can you create guide on ApScheduler python package
Apscheduler is great for windows, linux or mac. I created web app using flask+apscheduler,+angular for running py jobs, viewing logs.. everey job has its own py env using pipenv.
Do a Kaggle comp.
Rah rah rah! I like muffins!
Hey, can you please help me to schedule a job in python every 2 hours starting at some particular time. I wrote the code as - schedule.every(2).hours.at("01:00").do(func). but this is not working.
I recommend using one of the methods I described in this video. Hard to keep a python script actively executing for extended periods of time. Also if you do the method you described, you'll probably want to make sure the time zone is correct, because by default it will probably be UTC.
Do you use wsl 1 or wsl 2 ?
I think I use wsl 2 at this point, but to be honest I'm not even sure. Don't think the version should matter for the purpose of this video.
Great tips! What about running Python code on Android phones? There are schedulers but we're missing the first and foremost a Python interpreter here. :-( Given all the great Python content on your channel (so simple yet so powerful) how come there's nothing about running Python on the one platform out there capable of continuously generating huge amounts of invaluable data from all the sensors that smartphones have? Running ML models on real-time data in our pockets! Aren't Python+ML and smartphone sensors a match made in heaven???
Me and my 7 siblings are your subscribers!
Thank you!!!
yoyoyoyoyo yo! hahahaha
When I run the crontab -l or -e, I get the following text:
'Support/E3FE50D5-DB55-470F-8B7A-2B574AC40711/43139812-B903-4DD3-84F0-395806C84045 h >/dev/null 2>&1'
I've already gave full disk access to terminal but still haven't find anything on stackoverflow or other site to make crontab -l display the same. Unsure what this is :(
What OS are you on and what terminal are you using?
@@KeithGalli OS Catalina version 10.15.7 and the Mac terminal version 2.10
How to graphic design in Python tools
1) r'c:\full\path' == 'c:/full/path'
2) To get a full path og a file:
Hold shift + right mouse click on file -> "copy as path"
Useful tips, thanks! :)
@@KeithGalli No, Thank you.
Apricot
Install and Import schedule is much easier.
error:
I don't like WSL to start with. Too many pieces of Microsoft tools required. Not worth it if I have to use that.
any other suggestions?
@@EUU100 Sorry WSL is a trigger. I plunged into Linux Mint because I had Windows 10 crash for good on my Dell Inspiron. I'm comfortable with Debian packages and what I can run in docker images. Today I'm running Python in Jupyter Notebook against a MySQL database over a virtual network that maps client port to container port. I'd have to review the podcast here on schedulers and get back to you. If you can split boot right into Linux I'd do that over virtual Linux on Windows 10.
Would you like to partner with my company for merch
You talk to much to do a simple task
help!!!!
(base) huizi@huizideMacBook-Pro scheduling-code-master % Service cron status
zsh: command not found: Service