Відео

Nice video. VERY good writing and editing. Smooth as hell, keep it up!

I wonder how fast an AOT compiler like nuitka would make it

A 50x speedup between a for loop and a polars dataframe is really significant, great video!

Well... Your python code is not well "optimized" too =) 1. a, b - np.array, so you use numpy in your pure python code (that is not fair). Moreover, switching to lists gives you speedup (I'm shocked too). But, maybe you did so... You did not show us "all" code. 2. max of two elements is not faster than pure if-else. And if you use it - you need to extract dp[prev_i, j] and dp[i, prev_j] to variables to not duplicate accessing to elements. 3. Your i-1 and j-1 can be evaluated in corresponding loops and saved to variables. The same argument for len(a|b) + 1. Can be evaluated once. With all these I could speedup pure python up to +70% performance on my Ryzen 5 3600X. And it was consistant through N = 3_000, N = 10_000, so all my tests with numpy was with N = 3_000 to not waste too much time. Therefore optimized python code faster than not optimized numpy code by ~82.2%. If you apply 2) and 3) optimizations to numpy version, optimized python code faster by ~77.5%. I even tried different logic for numpy and python versions where dp is 1d array, but python version became slower, while numpy version made some poor progress (optimized numpy with different logic faster than optimized numpy by ~5%). Well, it was intresting. Never expirienced numpy code slower than pure python. Maybe because operations are so fast that switching to numpy types make it worse...

The video editing is top notch too!

Beautiful presentation. I love your descriptions on libraries. The illustrative code break down puts context and concrete examples on what would otherwise be yet more abstract documentation. It helps to grasp the utility of the libraries uou describe not just in this video but all of them. Fantastic work.

Why no c++ optimisation

What about pypy?

your video quality is top notch, im sure it will soon equate to video views if you keep this up, good luck.

As much as I like the video the 6 times slower not being a problem kinda drives me crazy as a person doing low level coding 😭 Like yeah you’re right for a lot of stuff it doesn’t matter, but then imagine 5 vs 30 fps, and then if it’s nested within another thing you may get exponentially slower. I have been trying to make a 480mhz mcus cpu do rendering and it’s difficult to get 30 fps ish and well if using python would mean 6 times slower I woudlnt be happy no matter how much easier my code would be to understand

should not have watched this video, considering the fact that I have been waiting two days for my implementation of kendal's Tau to finish,

That's nice of you to point these libraries out.

is pandas slow?

It's hard to watch this with all the whistles and blowers and another hangers, glitters and so on. Kindly, stop using those! The interesting video turns into a torture session!!

Damn, this was an instant follow! Hope to see more computer science content 🙏🏼 Great video :)

c++ -O3 flag: am i a joke to you?

python use third party compiler as well to execute fast.

1. "Some of these solutions are gonna be as fast or even faster than my C++ implementation" just means your C++ is bad lol. The python interpreter was built in C, so by definition a python program can never be faster than , or even as fast as C. C++ is SLIGHTLY slower than C, but python would never fall in between them. 2. Making python "faster" by transpiling it to C isn't making PYTHON faster, it's just converting it to C. It's basically like saying "I'm going to upgrade my Renault Twingo" and then getting a Bugatti Veyron

Kinda bummed I wasn’t sub 10k

I shit on Python a lot for being slow, but honestly, 8-10 seconds to read 1 billion rows is sufficient in most scenarios.

Depending on how large the total sum actually is, using an incremental mean may yield better performance since python won’t need to upgrade the number to a big int

are you allowed to use numpy or gpu (torch, cupy, etc)

I couldn't get your opening "performance critical python" out of my head and so missed the entire rest of the video.

I think it's 60x faster not 100x. We don't have 100 seconds in a minute. Great video though!

Pypy? Nuitka?

i would never use python, but i like watching how people optimize the hell out of something.

Beautifully done!

This is a beautiful explanation. Thank you for sharing it.

did you test out pandas to see how much slower it was than polars?

Can't really say you are just relying on python at that point. Well guess you can't say that to start with. Cpython runs on a JIT that is programmed in C to start with. If python is so great why don't they create a compiler with it to create a JIT that made from python that runs python.

How is this better than airflow?

8:53 I cannot believe writing a parser would gain any performance considering the default one is probably implemented in C. I assume this is a case of pypy optimizing while running and I'm wondering if running this same script with cpython would result in worse performance.

😮😅can you tell me how to measure how much time it take which part of code just by looking how ?

Nice channel. Hope it grows.

Instead of putting so much effort optimizing python, why not just use C/C++? Also multiple CPUs wont help, unless you have RAID0

Would it count as Python if we write it as a module in C?

Does this video's measurements get affected by Disk speed?

Me, approaching this as an engineer: - Read a random subset of the data - Do the computation on that - Yeah that's close enough lmao, interpolation will take care of missing values

"Dont write tests. Write test writers"

Looks like a python only alternative to Kestra - and still no sight of the software engineering practice to maintain build and ETL pipelines next to code in a CICD fashion… 😮‍💨

I've been using the pandas library, but it's given me a lot of trouble with a ~10GB dataset I have - so thanks for mentioning Polars and DuckDB, looks like both are ~10x faster with less memory usage. Tho I'm not sure if I appreciate SQL enough to go for DuckDB - I've been working with ClickHouse recently and even with the slowness of pandas, I end up spending way more time debugging my SQL queries. So that makes me feel like SQL is only worth the effort/time if it's actually going to be running often on the DB server, not for single-use processing scripts

Could you go into how you wrote the tests more?

How do you do the code animations?

Whats wrong with linux chronjobs ? if that malfunctions, the script is a non issue... since your OS will fail. as for the script, why do you call external functions raw like that in the main file, python itself has error handling and logging out of the box ! scheduling just another chronjob to mail reports can also be done .. but i don't like this, these "reports" can easily get out of hand if you don't have a centralized organized reporting system - pref nothing if nothing is wrong only if smt is wrong. The problem i have with this app is (1) if the startup goes down, or changes thier TOS, your company goes down with it - if a security bug was found after it went down your non-cloud stuff is also at risk (2) onboarding a new employee will be harder ; one more obscure framework for them to learn. ontop of all the company stuff. (3) IF stuff goes wrong don't just dump it into a file, or "prefex log" , get it fixed. if the logs are running out of hand, its symptoms of a bigger systems problem that you shouldn't just "patch up" . fix the code don't create extra procedure to be able to not fix it. Errors are your friends, silent failures are the worst. better read them, they will tell you whats wrong and even where :) (4) if the prefix lib needs to be updated, you need to update the code. if you never update it you risk (1) and also things like SSL change so it may work and suddenly not one day... its all extra work . (5) any time you add complexity, youre adding more points of potential failure. it would seem like a nightmare to me if prefix has an error... what if it cant parse some non uni-code characters or smt obscure you didnt think about... often keeping things simple is better. nice video though just some thoughts why im not convinced this is a good idea...just kicking the can down the road.

I dont know too much about C++, but I suspect that if some library is automatically making your python code faster than the C++ equivalent you very likely have a C++ skill issue

Talented but definitely need a senior engineer or a tech lead. There are a lot of ways to solve all of these problems (khm airflow, mlflow😂) Was this a real situation or just made up to help illustrate why this is in fact a great library?

You can't put "performance critical code" and "python" in one sentence

gcc -O3 -o subseq subseq.c