All those clickbaity websites and articles NEVER really talked about such a wonderful library that actually is a part of standard library. Today I learnt something new which will be actually helpful.
Performance tuning is usually an appendix in books and courses and the usual suggested solution is either timeit() or re-writing the critical part in c/c++. Thanks for showing a better approach.
This!! There's a book on Cython that right in the beginning shows how important it is to profile the code before attempting anything more elaborate, so as to not waste effort on something that isn't a bottleneck
@@dadestor What if you can reach something fast enough with some refactoring instead of rewriting in another language? Via profiling you can see quite easily if the program is spending a lot of time in functions that should not take much time, in which case you know what to refactor. Profiling can also help you see which parts of the program need to be particularly well written in the next language and which ones are not too critical on performance. Not exactly sure why you made your comment sound like you consider profiling as a waste of time.
Just to make it clear, what makes async so powerful is that it allows a cpu-python-process to parallelize waiting, in this case for responses from the network card. It is useful for data I/O to other devices, but it usually won't speed up stuff like costly operations on huge data-arrays. For that, multiprocessing is the way to go. If you happen to not like async for some reason, threading does basically the same thing, but as a different concept.
Thank you for the post, I didn't know of async, just multithreading and multiprocessing. Do you reckon it may be advantageous to use async when scheduling workers in a master/slave paralellization paradigm? Then again, I think OpenMP does this "by default" doesn't it? 🤔 (Bare in mind my understanding of these things is rather superficial, since OpenMP can be used in Cython via a fairly high level api)
@@bva0 Since the trend away from monolithic software to microservices is huge right now(and for good reason), whenever I hear keywords like master/slave or multiple workers, I always check if microservices make sense over classical concurrent processing. The additional networking overhead is usually neglible thanks to great tools like gRPC and once you have the architecture set up, scaling from one server to entire datacenters, the roadmap is suddenly just a (complicated) recipe you follow instead of a monstrous engineering problem. So if you run multiple slave workers anyway, I'd rather run more single process workers that are ephemeral instead of dealing with concurrency in monolithic software. There's the 1% of cases where this won't go well, e.g. training neural nets, but for those few cases there exist great compute frameworks already, e.g. tensorflow and pytorch. But sometimes the solution/problem is small enough and you really only want to parallelize waiting and/or are limited by some device I/O, then doing async or threading to do that can be a great choice for sure! HTH
@@THEMithrandir09 Thanks for the reply! I mainly use sparse direct solvers for optimization problems in mechanical engineering. Though I've been trying to migrate to sparse iterative solvers for HPC. I'll spend the next hour or so googling some terms in your answer, before asking further questions! Lol Edit: and yes, I use Tensorflow for some neural network problems! Though I find it difficult to fully understand the equivalence between certain numpy and tensor functions in order to fully get the benefits of operating in graph mode.
Thank you so much for this very timely video. I had an app that I presumed was slow because I had some inefficient SQL queries. After profiling, I discovered that the slowdown was actually in a regex that I was running against every returned record. I now only run the RegEx against records that contain certain text. It was taking 38 seconds to run. It now runs in 9 seconds, 8 of which is rendering the Jinja template. This video saved me what would have been many fruitless hours trying to improve the SQL, which turned out not to be the problem.
@@mCoding I think this would be a great video. I remember when I first started righting code that needed to be faster my instinct was just "well it's only using 1 core, so if I use all 4 cores it'll be 4 times faster!" Little did younger me know that instead of trying to orchestrate multiple cores all I needed to do was call some of my functions asynchronously to avoid IO bottlenecks.
Essentially, in the slow example, you're doing every get request one at a time, waiting for the first to finish before moving on to the next one. In the asynchronous example, you start all of the requests at the same time, doing them simultaneously, then wait for all them to finish before moving on.
I had never heard of anything related to this topic and actually, I had this problem numerous times when trying to do web scraping. That's exactly the problem with all tutorials on the internet. They only cover the basics and that's what makes this video so special for me.
The timing of this video is perfect, I need to profile a GUI that runs a little bit slow, and I have to investigate why :) Also I'm looking forward for more profiling videos, this could be a series
Awesome content. This literally comes as I had issues with time it took me to process large amounts of data. I spent hours trying to look up videos to speed certain operations up only to find it to have little impact. Can't wait to try this out.
Hey, I have a degree in CS and am employed as a Python dev. I fully understand threads and processes, but never use coroutines/ async-await. Would love an in-depth video in the differences in programming and usefulness, as well as what’s going on under the hood.
basically it relies on user-space context switching which is cheaper than kernel/thread-space context switching done in multithreading. you could do the same with multiple threads but overhead of switching thread contexts would be larger than event loop contexts.
This is fantastic. I've been working on a personal project that is sensitive to costly operations, and doing this has helped me significantly reduce runtime. No more guessing at where the problem is!
Pycharm also has a builtin run configuration called "profile code" or something like that, so you do not have to write the boilterplate cProfile code. This also shows a nice visualization
My code went from taking 47s to 8s then when changing the max concurent requests number I got it down to 4s then finally to 1.4s. 33x faster. Perfect video.
This is great info...good practical tips, presented to the point, with examples, and without all the time wasting filler content. Even IF it the purpose was to just make the video longer, the additional content was ACTUALLY useful. I'm sure there are plenty of videos on cProfile, but the bit about snakeviz was above and beyond. A lot of other channels that produce instructional videos can learn from this channel. I always look forward to more vids from this channel, even if they just randomly pop up on my stream. Time will spent!
Dude you're content is so refreshing, useful, to-the-point, and just simple superb. I've been bingeing your videos like crazy! Thank you and please keep it up 😊 P.S. quite surprise I haven't seen any comment about cumtime. Yes, very childish I know but I had to put it out there.
Great video! I'd also really like to see you do a whole video on how to best make use of asyncio. I roughly understand the API, but rarely know how to best make use of the different features. An introduction to your favorite async-powered libraries (e.g. httpx) would also be really helpful.
I had a good day today, but this new mCoding video just turned it into an ever better day! I didn't usually profile my code before this, but on my current project, I will. Thank you for the content!
Great presentation! Also you can achieve the same speed improvement with multi-threading or multi-processing with one request per thread or process. But that would mean longer video.
A python video that includes async that is not a) replace a print("Yes") with an async call to a function that calls print("Yes"), or b) Send requests to a webserver that takes exactly 1 second to reply by using socket and make the little change of completely rewriting the entire program to have it use async? In my lifetime? I'm SO going to hit that like button an odd number of times! Thank you! Also great to see snakeviz promoted, that was a lifesaver when i found it; If that's helpful for someone, here's a powershell function for calling the script with the profiler: Function pyprofile { python -m cProfile -o $env:TEMP\python_profiling.prof $args snakeviz $env:TEMP\python_profiling.prof }
Even though I knew just from the code that the web requests were the slowest part, I will definitely use the profiler in my future projects! Thx for the amazing video!
You are welcome! I know in this case the answer was obvious, but usually it's less obvious in a real project, I just wanted to keep it simple for this little video. Glad you enjoyed!
Yea like @Calf Dindon said each tasks will be run a bit and a "scheduler" has the role of switching which tasks has to be run. I believe it's when we call an `await` statement the scheduler will switch to another task and some kind of callback is set to run the logic once it can. If someone could confirmed this that would be great!
Imagine a teacher setting up kids for a test, but they want the kids to be in and out as fast as possible. So as kids start to line up she starts them on a test -> they start -> if one finishes the teacher goes to their test station -> writes down their score -> and sets up more kids for these tests. As you can see the teacher's only job is to start the kids on their test and not do it for them. Similar to the event loop. Python starts the requests and goes and starts more while waiting for the others to finish if that makes sense.
I quite like the profiling function in the Spyder IDE that comes with the Anaconda distribution. You can just hit F10 and it profiles your code and shows the output.
Another great video. Can you please make an elaborate video on async and parallel processing? It would be really helpful for people like me who don't have much understanding of such topics.
2:12 very very relatable I wish I knew about this a year ago when I was working on my thesis lol, this will be very useful, I always get to learn something new on this channel.
Could you make a video explanining asynchronous and parallel programming technically? I see many others just give a broad overview of how to use the library, not explain what it does under the hood. Great content!
This is on the horizon, but if you are looking for an existing video, David Beasley has a great 2 hour ish video that might go into the detail you are looking for.
Nice and informative video. Video coding tutorials are usually a waste of my time since reading text-based tutorials is faster, but somehow your videos are an Exception to the rule. BTW: at 8:34 how did you generate that list comprehension? What IDE is this?
For asynchronous HTTP requests, the two main go-to libraries are either this one (httpx) or aiohttp. But is there any speed difference or advantages using one library over the other? Or do they do the exact same thing and it's simply up to personal preference?
That's a question that you would have to answer yourself by reading the parts that you're actually using in a given project and/or profiling your code using both libraries.
Woah! Thanks a ton Any idea on profiling GUI or flask applications? In GUI we have different threads for GUI and one for behind the scene works. I need to profile the backend task threads
Opinion: JS has got its async/await much better & easier to use. This is the equivalent JS code for the htmls: const htmls = await Promise.all( urls.map(async (url) => { const res = await fefch(url); return await res.text(); }) );
And to answer "yes, but is this still slow?": think about how much data you're processing and how long it takes. That gives you the MBps, and THAT can indicate whether your code is still slow. If you have a super fast machine and super fast internet and the data is only a few megabytes, I would say it's still slow, because modern machines can handle gigabytes of data per second. Just a little rule of thumb to know whether your optimized code is still slow.
Was not aware of httpx, fully expected you to talk about aiohttp. Thanks for sharing this video it's really helpful. Would love to see your take on multiprocessing in python if you've not already done so.
The actual best, most unequivocal way to make your Python code faster... is reimplementing it in another lenguaje. Even better if you use efficient coding patterns rather than looks the most neat (aka most pythonic). Hurts but I know you can't argue otherwise.
Do note that this specific code block is *not* production ready - since it uses a blocking non-async reading of a file in an asynchronous environment ("with open(...)"). If for example that were to happen in an async web-server, that would be a horrific performance problem, making the whole server wait on that one read between all concurrent requests. Async in python is "colored functions" which is hard and sucks.
Funny thing is, i made a project that does the 3n1 check for a long range of number, and save the step and result in a file... I wrote in both c++ and python, surprisingly python took half as much time... i guess python is better in writing optimized c++ code than me😅😅
You really should have also shown profilehooks. It is just so much more convenient to add a decorator to a function call. It is easy to do and easy to remove then adding all the with ... blocks in the code. It has its limitations but to profile a single function at a time it is quite good and fast to use.
@@mCoding I actually did not realize that was a difference between the pro and free version. Makes sense. I'd go free as well if my job did not pay for my license. I also did not know about the asyncio lib... so that's also a handy tip. 👍Keep up the good work. Love your videos. It's nice to find more technical content... in my experience most python videos I see are either way to basic or a good level of complexity, but the explanations suck or the production quality is bad... so you hit a really nice balance.
@mCoding Love you videos and explanations! I'm a big fan and hope to see more of your content. Very interested in this example - however I notice that the resulting http/https counters are way higher that the synchronous function - many of the asynchronous responses are just empty - is this a bug or a feature? Would love to understand this better. Thanks in advance!
Ah found it - turns out the website URLs allmost all served a 301 redirect - fixed it by adding `follow_redirects=True` to the client.get line, so the full line is: tasks = (client.get(url, follow_redirects=True) for url in urls)
Thank you so much for your videos, by the way , Can you tell me which software did you use to record the video? How to put the speaker in the video， I also want to use. Thank you so much.
How do you get to know these stuff ? Suggestions for resources where I can find them ( apart from open source projects ). Something like good blogs etc. Where they break down things a little for you effectively ?
I have a simular situation. My intention is to get links from a webpages and download the files from said links. My solution is to use a loop to browse through webpages and get the links, then download them. Would it be better to write all the links to a file first then downloading it?
It doesn't matter much whether you write the links to a file (though you may want to do that for your records), the important part is that you download the files asynchronously just like we download the pages themselves asynchronously in this video. Just keep in mind that automated downloading of files may be against the terms of service of a site even if you would have access to download all of the files by hand.
All those clickbaity websites and articles NEVER really talked about such a wonderful library that actually is a part of standard library. Today I learnt something new which will be actually helpful.
Thank you so much! Glad you enjoyed!
Performance tuning is usually an appendix in books and courses and the usual suggested solution is either timeit() or re-writing the critical part in c/c++. Thanks for showing a better approach.
This!! There's a book on Cython that right in the beginning shows how important it is to profile the code before attempting anything more elaborate, so as to not waste effort on something that isn't a bottleneck
well profiling is basically timing and if you need high speed python won't cut it anyways no matter what you do with it
@@dadestor What if you can reach something fast enough with some refactoring instead of rewriting in another language? Via profiling you can see quite easily if the program is spending a lot of time in functions that should not take much time, in which case you know what to refactor.
Profiling can also help you see which parts of the program need to be particularly well written in the next language and which ones are not too critical on performance.
Not exactly sure why you made your comment sound like you consider profiling as a waste of time.
@Andrii Shafar v8 is good, but it's still an interpreted (or JIT -ed) language
I never expected to learn such a thing, this channel is full of stuff that i never expect to learn, thanks man :)
Happy to hear that!
exactly, that's why I love this channel, lots of beginner content out there, but not as much intermediate/advanced stuff and that's what we get here
Just to make it clear, what makes async so powerful is that it allows a cpu-python-process to parallelize waiting, in this case for responses from the network card. It is useful for data I/O to other devices, but it usually won't speed up stuff like costly operations on huge data-arrays. For that, multiprocessing is the way to go. If you happen to not like async for some reason, threading does basically the same thing, but as a different concept.
Thank you for the post, I didn't know of async, just multithreading and multiprocessing.
Do you reckon it may be advantageous to use async when scheduling workers in a master/slave paralellization paradigm? Then again, I think OpenMP does this "by default" doesn't it? 🤔
(Bare in mind my understanding of these things is rather superficial, since OpenMP can be used in Cython via a fairly high level api)
@@bva0 Since the trend away from monolithic software to microservices is huge right now(and for good reason), whenever I hear keywords like master/slave or multiple workers, I always check if microservices make sense over classical concurrent processing. The additional networking overhead is usually neglible thanks to great tools like gRPC and once you have the architecture set up, scaling from one server to entire datacenters, the roadmap is suddenly just a (complicated) recipe you follow instead of a monstrous engineering problem.
So if you run multiple slave workers anyway, I'd rather run more single process workers that are ephemeral instead of dealing with concurrency in monolithic software.
There's the 1% of cases where this won't go well, e.g. training neural nets, but for those few cases there exist great compute frameworks already, e.g. tensorflow and pytorch.
But sometimes the solution/problem is small enough and you really only want to parallelize waiting and/or are limited by some device I/O, then doing async or threading to do that can be a great choice for sure!
HTH
@@THEMithrandir09 Thanks for the reply! I mainly use sparse direct solvers for optimization problems in mechanical engineering. Though I've been trying to migrate to sparse iterative solvers for HPC.
I'll spend the next hour or so googling some terms in your answer, before asking further questions! Lol
Edit: and yes, I use Tensorflow for some neural network problems! Though I find it difficult to fully understand the equivalence between certain numpy and tensor functions in order to fully get the benefits of operating in graph mode.
I'm thinking of having a "when to use async vs multithreading vs multiprocessing vs nothing" video but first I need to introduce all the topics :)
@@mCoding Thats great idea!
Thank you so much for this very timely video. I had an app that I presumed was slow because I had some inefficient SQL queries. After profiling, I discovered that the slowdown was actually in a regex that I was running against every returned record. I now only run the RegEx against records that contain certain text. It was taking 38 seconds to run. It now runs in 9 seconds, 8 of which is rendering the Jinja template. This video saved me what would have been many fruitless hours trying to improve the SQL, which turned out not to be the problem.
So great to hear your success story! I'm glad my video helped!
Could you make a video going in more depth on the difference between async and parallel coding?
Planning this sometime in the future, we'll see how long it takes to make...
@@mCoding I think this would be a great video. I remember when I first started righting code that needed to be faster my instinct was just "well it's only using 1 core, so if I use all 4 cores it'll be 4 times faster!" Little did younger me know that instead of trying to orchestrate multiple cores all I needed to do was call some of my functions asynchronously to avoid IO bottlenecks.
yes please!
@@mCoding please talk about gevent
wtf. even the plain words scream the difference in my face. the difference is so obvious. dont know why its even a topic worth.
Essentially, in the slow example, you're doing every get request one at a time, waiting for the first to finish before moving on to the next one. In the asynchronous example, you start all of the requests at the same time, doing them simultaneously, then wait for all them to finish before moving on.
I was using async requests when needed but the profiling tool is a game changer, that was something I did not know existed.
OMG I've landed on a gold mine.
This channel is awesome 👍👍👍
Thank you so much I'm glad you enjoy!
I had never heard of anything related to this topic and actually, I had this problem numerous times when trying to do web scraping. That's exactly the problem with all tutorials on the internet. They only cover the basics and that's what makes this video so special for me.
Keep it up, great quality content!
Thanks, will do!
Can you make more videos on intermediate topics? This is my first time learning about async and it already seems so helpful!
Your video quality just keep rising!
The timing of this video is perfect, I need to profile a GUI that runs a little bit slow, and I have to investigate why :)
Also I'm looking forward for more profiling videos, this could be a series
More to come!
Icicle graph … flame graph ! Wow, python makes that easy. What a wonderful developer experience. It even serves it in your browser for you 😳😱😍
Awesome content. This literally comes as I had issues with time it took me to process large amounts of data. I spent hours trying to look up videos to speed certain operations up only to find it to have little impact. Can't wait to try this out.
Been using python 2 years and never knew it could do async/await. Thanks for this!
Hey, I have a degree in CS and am employed as a Python dev. I fully understand threads and processes, but never use coroutines/ async-await. Would love an in-depth video in the differences in programming and usefulness, as well as what’s going on under the hood.
basically it relies on user-space context switching which is cheaper than kernel/thread-space context switching done in multithreading. you could do the same with multiple threads but overhead of switching thread contexts would be larger than event loop contexts.
This was a terrific video - I just cut a script from 45s to 3s using your technique. Thank you James!
Awesome! Great to hear and glad that you were able to use snakeviz to go use!
I prefer kernprof which can shows your code's execution speed line-by-line which can really help in some situations
Cool, first time I read about this
This is fantastic. I've been working on a personal project that is sensitive to costly operations, and doing this has helped me significantly reduce runtime. No more guessing at where the problem is!
Great to hear!
In a world of beginner tutorial, here comes a saviour with intermediate topics. Thanks a ton mate
Happy to help!
I was struggling to find good, simple but practical examples of the advantages of async and await but this has really cleared it up for me.
Pycharm also has a builtin run configuration called "profile code" or something like that, so you do not have to write the boilterplate cProfile code. This also shows a nice visualization
My code went from taking 47s to 8s then when changing the max concurent requests number I got it down to 4s then finally to 1.4s.
33x faster.
Perfect video.
Your process of profiling is a big improvement to iterating through profiling output. Very helpful. Thank you for your video tutorial!
You're very welcome!
Wonderful! Thank you so very much for this introduction to profiling.
You're very welcome!
Excellent video! Profiling is a super important tool! Your explanation and examples were very clear and useful!
Thanks, am a new programmer learning
Happy to help!
This is great info...good practical tips, presented to the point, with examples, and without all the time wasting filler content. Even IF it the purpose was to just make the video longer, the additional content was ACTUALLY useful. I'm sure there are plenty of videos on cProfile, but the bit about snakeviz was above and beyond. A lot of other channels that produce instructional videos can learn from this channel.
I always look forward to more vids from this channel, even if they just randomly pop up on my stream. Time will spent!
Thanks for the kind words! Glad you enjoyed!
Today I learned that if you delete the 2 second sleep it takes 2 seconds off the time...
Seriously though, great video, thanks :)
You'd be suprised how easy it it is for a debug sleep or print statement to end up in a prod release ;)
Dude you're content is so refreshing, useful, to-the-point, and just simple superb. I've been bingeing your videos like crazy! Thank you and please keep it up 😊
P.S. quite surprise I haven't seen any comment about cumtime. Yes, very childish I know but I had to put it out there.
More on async in python please!
Great video! I'd also really like to see you do a whole video on how to best make use of asyncio. I roughly understand the API, but rarely know how to best make use of the different features. An introduction to your favorite async-powered libraries (e.g. httpx) would also be really helpful.
I had a good day today, but this new mCoding video just turned it into an ever better day! I didn't usually profile my code before this, but on my current project, I will. Thank you for the content!
Not gonna lie, expected the "feat. async/await" to be "you are using async where you didn't need to and that is slowing you down" :p
I really appreciate the fact you skipped the running time at 02:00
Thank you so much for contributing to the community in such a wonderful way!
Great presentation! Also you can achieve the same speed improvement with multi-threading or multi-processing with one request per thread or process. But that would mean longer video.
Maybe a topic for a future video for sure! Thanks!
A python video that includes async that is not a) replace a print("Yes") with an async call to a function that calls print("Yes"), or b) Send requests to a webserver that takes exactly 1 second to reply by using socket and make the little change of completely rewriting the entire program to have it use async? In my lifetime? I'm SO going to hit that like button an odd number of times! Thank you!
Also great to see snakeviz promoted, that was a lifesaver when i found it; If that's helpful for someone, here's a powershell function for calling the script with the profiler:
Function pyprofile
{
python -m cProfile -o $env:TEMP\python_profiling.prof $args
snakeviz $env:TEMP\python_profiling.prof
}
Masterpiece! Thanks for your effort, I managed to get my hands to keyboard from other side of sofa just to write this comment.
Even though I knew just from the code that the web requests were the slowest part, I will definitely use the profiler in my future projects! Thx for the amazing video!
You are welcome! I know in this case the answer was obvious, but usually it's less obvious in a real project, I just wanted to keep it simple for this little video. Glad you enjoyed!
Amazing, but even though it’s single threaded how does it handle the whole asynchronous part?
Python magic behind the scene, i think that it gives each task a small cpu time and switches to the next task.
Yea like @Calf Dindon said each tasks will be run a bit and a "scheduler" has the role of switching which tasks has to be run.
I believe it's when we call an `await` statement the scheduler will switch to another task and some kind of callback is set to run the logic once it can.
If someone could confirmed this that would be great!
You can read about the event loop design pattern here! en.wikipedia.org/wiki/Event_loop
Imagine a teacher setting up kids for a test, but they want the kids to be in and out as fast as possible. So as kids start to line up she starts them on a test -> they start -> if one finishes the teacher goes to their test station -> writes down their score -> and sets up more kids for these tests. As you can see the teacher's only job is to start the kids on their test and not do it for them. Similar to the event loop. Python starts the requests and goes and starts more while waiting for the others to finish if that makes sense.
Yeah thank you everyone, makes a lot of sense. Didn’t realize it was like an event handler of some sort .
Holy fuck, dude! How do you come up with all these ideas for new videos?! I so enjoy your content! Please keep up the good work! ;)
so many interesting videos to learn things here and there, thanks a lot
Advanced stuff with so much new stuff to learn about python modules.
15 Iterations with a two second sleep between each one in 12 seconds!! Man, this guy is really fast!!!
I believe the sleep was outside the loop!
I quite like the profiling function in the Spyder IDE that comes with the Anaconda distribution. You can just hit F10 and it profiles your code and shows the output.
Thank you for making such a high quality video. Amazing.
Man!
You create amazing content! Thanks for all.
Nice tricks you put here. Thanks a lot.
Another great video. Can you please make an elaborate video on async and parallel processing? It would be really helpful for people like me who don't have much understanding of such topics.
It's been in the works for a while... hard to get right!
i learned alot this video yet again, thank you so much!
You're very welcome!
2:12 very very relatable
I wish I knew about this a year ago when I was working on my thesis lol, this will be very useful, I always get to learn something new on this channel.
There's always a next time!
line_profiler is nice - and way more intuitive to use than a flamegraph.
if you made a course I would buy it, you are amazing!
I’ve been wanting to know how to profile my Python code! This is SUPER helpful!
Could you make a video explanining asynchronous and parallel programming technically? I see many others just give a broad overview of how to use the library, not explain what it does under the hood. Great content!
This is on the horizon, but if you are looking for an existing video, David Beasley has a great 2 hour ish video that might go into the detail you are looking for.
Nice and informative video. Video coding tutorials are usually a waste of my time since reading text-based tutorials is faster, but somehow your videos are an Exception to the rule.
BTW: at 8:34 how did you generate that list comprehension? What IDE is this?
he probably had it copied and just pasted the list comprehension
For asynchronous HTTP requests, the two main go-to libraries are either this one (httpx) or aiohttp. But is there any speed difference or advantages using one library over the other? Or do they do the exact same thing and it's simply up to personal preference?
That's a question that you would have to answer yourself by reading the parts that you're actually using in a given project and/or profiling your code using both libraries.
Woah! Thanks a ton
Any idea on profiling GUI or flask applications?
In GUI we have different threads for GUI and one for behind the scene works. I need to profile the backend task threads
This is brilliant, thank you for sharing this. The profiling is going to be really useful at work.
Extremely helpful and goes straight to the point. This type of content is very, VERY valuable!!!! Thank you!!!
You are very welcome I'm glad you enjoyed!
Amazing content and very well delivered, thanks mate!
Thanks so much!
You’re amazing. Keep the good work.
Thank you very much for the kinds words!
This is a great video. Wow. Thanks!
Thank you for your kind words!
I liked your videos about how to make python run faster!
Opinion: JS has got its async/await much better & easier to use.
This is the equivalent JS code for the htmls:
const htmls = await Promise.all(
urls.map(async (url) => {
const res = await fefch(url);
return await res.text();
})
);
And to answer "yes, but is this still slow?": think about how much data you're processing and how long it takes. That gives you the MBps, and THAT can indicate whether your code is still slow. If you have a super fast machine and super fast internet and the data is only a few megabytes, I would say it's still slow, because modern machines can handle gigabytes of data per second.
Just a little rule of thumb to know whether your optimized code is still slow.
Was not aware of httpx, fully expected you to talk about aiohttp. Thanks for sharing this video it's really helpful. Would love to see your take on multiprocessing in python if you've not already done so.
My pleasure! Been working on an async/threading/multiprocessing video for a while, it's hard to get right!
The actual best, most unequivocal way to make your Python code faster... is reimplementing it in another lenguaje. Even better if you use efficient coding patterns rather than looks the most neat (aka most pythonic). Hurts but I know you can't argue otherwise.
Lol
Very useful and nice clean examples. Thank you.
You are welcome!
Amazing and straight to the point. Thanks
"Here's the random sleep someone forgot to delete. We can just delete that."
X doubt. I would instantly test it to see if it still works.
Very clear and detailed explanation! Thank you very much!!!
You're welcome and thanks for watching!
That looks very useful, ill definitely use it when I need to.
This was really helpful
Awesome, glad you think so!
snakeviz is awesome! Thanks James!
You bet! Thanks for watching again!
Your channel is amazing! thanks a lot for such excellent and useful content!
Do note that this specific code block is *not* production ready - since it uses a blocking non-async reading of a file in an asynchronous environment ("with open(...)"). If for example that were to happen in an async web-server, that would be a horrific performance problem, making the whole server wait on that one read between all concurrent requests. Async in python is "colored functions" which is hard and sucks.
Liked and subscribed!! That was helpful
Great to hear! Welcome to the community!
Httpx ftw!
I know, I can't wait for it to hit 1.0!
Great focus on the task, but the non pythonic sums drive me crazy :D
Keep up the great content mate!
Really helpful. Cheers!
Thanks, will do!
AMAZING! Thanks for sharing!
You are very welcome! Glad you enjoyed!
this was really helpful, thank you!
Glad it was helpful!
Brilliant! Thank you.
Funny thing is, i made a project that does the 3n1 check for a long range of number, and save the step and result in a file... I wrote in both c++ and python, surprisingly python took half as much time... i guess python is better in writing optimized c++ code than me😅😅
beautiful! i like the content of this channel.
You really should have also shown profilehooks. It is just so much more convenient to add a decorator to a function call. It is easy to do and easy to remove then adding all the with ... blocks in the code. It has its limitations but to profile a single function at a time it is quite good and fast to use.
Yet another fantastic video. Thanks for this. I learned a lot! Do you ever use the profiler in PyCharm?
I use the free version of PyCharm :S so no profiler for me.
@@mCoding I actually did not realize that was a difference between the pro and free version. Makes sense. I'd go free as well if my job did not pay for my license. I also did not know about the asyncio lib... so that's also a handy tip. 👍Keep up the good work. Love your videos. It's nice to find more technical content... in my experience most python videos I see are either way to basic or a good level of complexity, but the explanations suck or the production quality is bad... so you hit a really nice balance.
Excellent and very interesting
How about benchmarking the same task done with Perl 5?
@mCoding Love you videos and explanations! I'm a big fan and hope to see more of your content. Very interested in this example - however I notice that the resulting http/https counters are way higher that the synchronous function - many of the asynchronous responses are just empty - is this a bug or a feature? Would love to understand this better. Thanks in advance!
Ah found it - turns out the website URLs allmost all served a 301 redirect - fixed it by adding `follow_redirects=True` to the client.get line, so the full line is:
tasks = (client.get(url, follow_redirects=True) for url in urls)
Thank you so much for your videos, by the way , Can you tell me which software did you use to record the video? How to put the speaker in the video， I also want to use. Thank you so much.
I'm using obs studio to record, and I have a grewn screen behind me so I can use the chroma key filter to make the background transparent.
How do you get to know these stuff ? Suggestions for resources where I can find them ( apart from open source projects ). Something like good blogs etc. Where they break down things a little for you effectively ?
I love it! Just love it!
Like + Fav on it!
Thanks so much 🙏
KUDOS!! Sincerely, I want you to thank you for your great videos!! I quite appreciate the knowledge you're sharing with us
Thank you very much!
Really deepdiving into coroutines would be great
I have a simular situation.
My intention is to get links from a webpages and download the files from said links.
My solution is to use a loop to browse through webpages and get the links, then download them.
Would it be better to write all the links to a file first then downloading it?
It doesn't matter much whether you write the links to a file (though you may want to do that for your records), the important part is that you download the files asynchronously just like we download the pages themselves asynchronously in this video. Just keep in mind that automated downloading of files may be against the terms of service of a site even if you would have access to download all of the files by hand.