super/MRO, Python's most misunderstood feature.
Вставка
- Опубліковано 19 чер 2024
- Python's super does NOT mean "parent".
It means "next in line". What line? The Method Resolution Order (MRO) of an object, which defines the search order for attribute lookups. super() uses some very sneaky techniques, such as examining the current stack frame, and object proxying. All this is explained in great detail, for both single and multiple inheritance, and a pure Python implementation of super is given.
CONTEST CURRENTLY CLOSED! THE WINNERS HAVE BEEN CONTACTED, CHECK YOUR COMMENT TO SEE!
OFFICIAL CONTEST RULES:
1. All entries must comply with the UA-cam community guidelines ( ua-cam.com/users/tcommunity_gu...) and UA-cam Terms of Service (ua-cam.com/users/static?gl=US&t.... Entries that violate UA-cam guidelines are automatically disqualified.
2. UA-cam is not a sponsor of the contest and viewers are required to release UA-cam from any liability related to the contest.
3. Privacy notice: no personal data will be collected for this contest.
4. In order to enter, you must (a) be one of my subscribers, AND (b) make a top-level comment to the video including #pycharm or #clion somewhere in the comment.
5. The contest is free, there is no fee required to enter.
6. Winners will be chosen randomly 1 week after the date the video went live from all users who have entered and not been disqualified.
7. Each winner will be notified via a comment reply from me that details what prize was won (e.g. "Congratulations! You have won XYZ. Please email me."). I will ask the winner to contact me by email, and I will reply through email with a random token which must be posted as another reply to the winning comment from the winning account in order to verify account ownership and prevent fraud.
8. Each winner will have 72 hours to respond AND prove account ownership or their prize is automatically forfeited and another winner will be chosen.
9. A winner can only win 1 prize per contest.
10. The prize pool for this contest is: 4 licenses for PyCharm Professional and 5 licenses for CLion ("Free 1-Year Personal Subscription"), which are products made by JetBrains. A prize consists of 1 license, which will be delivered in the form of a redeemable code that can be redeemed at www.jetbrains.com/store/redeem/ before May 01, 2022.
11. You may not enter the contest if doing so would be a violation of any relevant federal, state, and local laws, rules, and regulations, including U.S. sanctions.
― mCoding with James Murphy (mcoding.io)
Source code: github.com/mCodingLLC/VideosS...
super docs: docs.python.org/3.8/library/f...
A super guide: rhettinger.wordpress.com/2011...
SUPPORT ME ⭐
---------------------------------------------------
Patreon: / mcoding
Paypal: www.paypal.com/donate/?hosted...
Other donations: mcoding.io/donate
Top patrons and donors: Jameson, Laura M, Dragos C, Vahnekie, John Martin, Casey G
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding
CHAPTERS
---------------------------------------------------
0:00 Intro
0:32 Basic super usage
2:34 super does NOT call your parent
4:29 Method Resolution Order (MRO)
6:18 super means "next in line"
6:48 How MRO is determined
8:07 How to design with super
12:01 How does Python super work?
16:53 Two-argument super
18:01 One-argument super
18:18 Implementing super in Python - Наука та технологія
Isn't the search that the MRO is doing the breadth-first search?
In this case it was the same as a BFS, but in general it does not have to be! See the L(Z) example in en.wikipedia.org/wiki/C3_linearization for a case where it is not a BFS.
@@mCoding my assumptions keep breaking and I love it.
Technically, it does a depth-first, left-right search and can contain duplicates. Then, it removes all but one of the duplicates. The last duplicate is preserved, all else are removed.
Eg:
class A: pass
class B(A): pass
class C(A): pass
class D(B, C): pass
so depth-first, left-right search list for 'D' would be something like this:
['D', 'B', 'A', 'object', 'C', 'A', 'object']
Now, all earlier duplicates will be removed for the __mro__, i.e the first 'A', and the first 'object' will be removed from list.
['D', 'B', 'A'(first 'A', hence removed), 'object'(first 'object', hence removed), 'C', 'A', 'object']
the final list is this:
['D', 'B', 'C', 'A', 'object']
@@mCodingthat algorithm looks like an O(n^3) mess lol
I've been programming in Python professionally for almost 10 years and I learned something in this video.
Glad to hear!
Same!
does that mean these things are not going to be used in industry?
@@TemporaryForstudy not if you want to keep working in that industry ;-)
@@Ratimus_ Thanks for the information. Are you a python developer?
i spent days treading in convoluted documentation to finally understand MRO, multiple inheritance, and super (had to also get through descriptors to understand the bounding mechanics)... and here it is, a 20 minute video that explains everything. this channel is way too underrated.
Raymond Hettinger's old talk "Super is considered super" is fairly good, too
Thanks for the kind words! And I also recommend "Super is considered super"! It's linked and Hettinger was (I believe) the one who coined the "next in line" phrase.
@@mCoding If anything, it's worth watching for the volunteer part (explaining cooperative inheritance and MRO) at very least. Visuals like that make things a lot easier to comprehend/memorize. I'm not a fan of some of Raymond's teaching techniques but this specific part was brilliant.
With that experience maybe you could make the documentation better? I’m not trying to be one of the “then open a pull request!” guys but it sounds like you have special insight that might help future users.
The *level* of this channel is 👌. I've been coding in Python for quite awhile and I always learn stuff here. It's really refreshing when a lot of what is out there is like "how to use dicts in Python!"
yeah that the problem, he is teaching super , to advanced people.
So true, it's high level, but it doesn't use an overwhelming amount of jargon. Learning a new programming language is a pain, because 90% of the material and tutorials are "This is the syntax for a for loop", while the remaining 10% is an argument on stack exchange about the best boosting the efficiency of a highly specific package from 1994 with some compiler meta-optimization. It's either teaching a complete beginner how to code for the first time, or a set of code-words that will remind an expert of how to do something. When I almost always want is help on turning my ok-ish code, into good code.
I wish he did material on, say, C# or Julia, so I could go from a basic understanding of the surface feature of the language to having an idea of how the structure of it works.
Exactly! And it's always some of the most error-prone structures that you can't understand!
Love your videos!
James: your expert knowledge of not just Python, but C, and CPython implementation - along with dense and info-rich presentation - make for a quite rare resource of deep, under-the-hood understanding of Python. Thanks for putting in the work!
This is why multiple inheritance, especially "diamond" multiple inheritance is evil. Even if you're one of the 0.01% who knows how it works, it will still trip you up when you dont expect it (that is, 99% of the time). Having recently come across C#'s solution to this (a class can only inherit from 1 class, but can inherit from as many Interfaces as it wants), I now think python's approach is too liberal. Of course that's the whole python ethos, but sometimes it creates far more traps than necessary. #pycharm
Too many people take my job security comment a bit too seriously!
Here the trap isn't that problematic imo, once you inherit from multiple classes there is an obvious problem with super.
Best practise I think is to specify which parent function you want to call
So you've discovered interfaces. My favorite C# feature. Initially they seem hard but once you get the hang of it you'll never go back to MI. When you get a bit deeper you'll hit covariance / contravariance with interfaces, and the slightly ugly type parameter constraints that all kind of tie together. C#'s type system is not quite perfection.
That's precisely the case with Java as well.
Don't all classes derive from object, making any multiple inheritance diamond shaped?
Thanks for being so clean and straight-to-the-point in how you present your topics. Your videos are quite like 3Blue1Brown but for programming. Keep up the dry humor, stellar stuff. #pycharm
The 2 argument super used to be the required way of calling super in python2. So glad we don't need that anymore
May Python 2 rest in peace, forever!
Amazing insights into the super() function implementation in Python! Thank you for your great work, I always learn something new!
Glad it was helpful!
The only time I used multiple inheritance it was a big mistake, it became impossible to have the classes cooperate nicely. A slightly more verbose solution based on composition solved the same problem without all the subtle bugs
#pycharm
As is always a question you should ask, just because you *can* solve the problem with multiple inheritance, *should* you?
@@mCoding I believe 99,99999% of the time the ansewer is sounding NO.
Rule 2 of OOP: Favor Object composition over inheritance
@@mCoding Could you please make a video where you show with a few different examples to replace a solution with inheritance with functional composition?
Very nice! Your example code at the end is the part that was missing in PEP 3135 back when it was written, it would have cleared up how super works for most people. For a good example, look at PEP 343 - with Statement, where the "Specification" section shows exactly how it works in a very clear fashion.
Even though I am aware about using super and cooperative multiple inheritance when I watched the iconic talk of Mr. Raymond Hettinger on super()...
But even then, my mind was simply in awe when you dived into the details... I never even imagined that super class is sort of implementing a Proxy pattern...
I love these metaprogramming videos. The best way to explain a complex language features is to show the implementation.
Fantastic introduction to super() and MRO, thanks.
My pleasure! Have a super day!
Another great video, the first half was satisfyingly confirming some practices I have picked up by using python for a while, and the latter half was part mind-boggling, part "oh please don't do that". Learned a lot as always, thank you James! #pycharm
Glad you enjoyed it!
A good addition for cooperative inheritance is that the root class' sink method verifies that both *args and **kwargs are empty at that point and throws an error if either isn't. This helps prevent unnecessary arguments being passed somewhere in the chain that stick around after changes/refactoring as the *args and **kwargs in the signatures of children mean that static code analysis often won't see anything wrong.
Love the 1-letter variables in your functions. Right after coming from one of your "nooby habits" videos too.
If someone says that python is easy, sure they didn't saw this video, it's a good example of how much python has to offer below the surface. Very helpful! Thanks!
This comment applies to the metaclasses one too. That one blew my mind the first time
@@sebastianandrade8500 Let's not forget descriptors and __getattribute__(). Python really does a good job in hiding complexity. It's so simple yet so expressive.
IIRC, you can rebind super to have the compiler magic keep working while also allowing you to change its behaviour (it's definitely the case that using super bound to another name causes the compiler magic to fail)
EDIT: To be clear, if you had written `class super:` instead of `class Super:` it would be a drop-in replacement
This channel is pure gem!
Wow - that was complicated but enlightening. Your videos are great!
13:18 little mistake. __getattr__ is called whenever getting attribute is failed, __getattribute__ is called everytime you trying to get an attribute.
In current example it is the right form of using dunder getattr method. Since you don't want to get the attribute of an instance of this class but the self.obj, that will make the code work excatly as it should. Changing dunder getattr to getattribute will cause the RecusrionError because it will invoke itself infenetily long, as you're trying to get the attribute not of self.obj but of self - an instance of current class.
Intermediate programming channel are so rare.
It's always a pleasure watching your videos !
#clion
This was the most I have learned about python in any 20 min ever.
Subscribed
And here I thought “How complicated can super() really be?” It’s a joy to learn from you. Keep up the great content! #pycharm
It's a bit scary how much work at runtime goes behind a simple attribute/method lookup.
To be fair, it's python, which is about convenience over speed. If you want method and property lookups to be fast, you should probably be using a different kind of language
@@berylliosis5250 Considering that attribute/method lookups are one of the most common things you do in a programming language, it's quite frightening that what takes one machine instruction in a compiled language takes upwards of 100 *bytecode* instructions, including both branches and memory fetches.
I hope (and think, but am too lazy now to check) that Python does some optimizations here to bypass this cost because I've never really felt that Python is that slow in comparison. I do expect a certain level of speed from any language.
I would love to see an explanation of the Walrus (:=) operator. How it works,tips and tricks, and use case scenarios. Mostly I've missunderstood when i should/can use it. My code often feels non-reable when using it.
Loved this video (and ofc your previous ones)!
#pycharm
I remember that I watched Raymond Hettinger's "Super considered super!", understood a little more about the MRO but could not understand how `super()` works.
Now I have watched this video and I have learnt something new today! Thank you!
I still don't fully understand the `super(...)` with argument(s) form, but I guess/hope I don't need it since I don't need to consider my job security. 😂
Amazing. Your videos are so consistently great. Good work!
As a C/C++ dev, all this runtime stuff python is doing just to do basic static things is horrifying.
As a Rust dev, all this unsafe UB in C/C++ is horrifying! :D
@@josephlyons3393 At least we don't have a whiny compiler. :)
Just kidding, I like the rust's approach. Cppckeck for cpp is sometimes all what you need tho.
@@embeddor3023Rustc certainly is soul crushing, lol. C++ was my main language for many years, but I mostly dev in either Rust or Python these days, depending on the task.
I find it quite enlightening when you dive deeper into how these features work rather than working with the superficial layer of when to use/ how to use these features. I always come away with some useful knowledge I can apply to my code. #pycharm
I love the level of depth you manage to go into in such a short time. Few guys on UA-cam deliver this stuff as well as you do. Also #pycharm :)
Wanted to get into open source dev. One of the main projects I wanted to get into is pretty #clion oriented would love a key!
Finally, a video that covers the inner details of super and how it works, also the pure python implementation of super is awesome, learnt a lot from this video, thank you so much Murphy. #pycharm
Very welcome and glad you enjoyed!
You are really good at teaching. I like how you build up the complexity. Nice!
Wow. You put in a LOT of detective work on this. Thanks.
Object oriented programming in Python is complicated. Thank you for this video!
Well, tbh. in most cases it's not a big deal because you can compose functions pretty easily by explicitly calling them. You don't need "object oriented" programming (more like inheritance) in these situations.
It seems like it, but it's actually far less complicated _in practice_ than in many other languages, like C++ (public vs private inheritance, recursive inheritance [CRTP], pure abstract classes, constness concerns, public/protected/private members, etc. -- and that's before we add templating!).
In both cases, though, you should really avoid inheritance for code reuse, anyway. Inheritance should be used for polymorphism. Only in some rare cases when following certain design patterns (or idioms) should you inherit anything but the interface, and in those cases, you should just follow the pattern. Generic programming is usually closer to what you want with code reuse; Python has this with duck typing, while languages like C++ do this with templates, and some others like Java fake it (somewhat poorly) with type erasure.
Discord gang 🤙🤙
Always appreciated!
I am glad I came across your channel. Just realised I've been more focused on the 'how' things work in python but didn't give much thought to the 'why'...good job mate. #pycharm
Awesome explanation, especially the part on how super works underneath the hood!! Thanks a lot.
Note to self: never use inheritance in Python if I want my code to run fast. Seeing what goes on behind the scenes is amazing, and a bit scary... the VM must be doing *so* much work every time a super() gets called XD
after like 8 years of programming I've come to the conclusion to use inheritance as little as possible it just creates a mess in places you never think of
It is impressive, how Python implements all these functionality _with Python_ - from a technological point of view. Python, as an interpreter language, implements itself by using Python... fascinating!
But from the perspective of a user of the language ... all this black magic everywhere ... meh. :/ In other languages, using "clever coding" is regarded as something bad ...
I'm assuming that super is implemented in C (at least for cpython) and not in pure python since it's a built-in. But it could be implemented in python, aside from the class injection hack.
Compared to most other languages, python has very little black magic. Other languages also have things like implicit `this` variable, which feels more like black magic to me
I think this is a case of "practicality beats purity" :D
I remember programming in Python 2, which didn't have this black magic. This meant that every. single. super call needed to be the two-argument form... Believe me, it gets old very quickly, apart from being error-prone, as you now have to mention the class you're in everywhere.
Oh my God thank you. Having a video this concise and clear is really really helpful.
I have been wondering whether a function like this exists for a long time. Thank you!
Very welcome!
I've been asked many times about a clear explanation of super(), and here it is!
Your python knowledge is truly frightening... I like the video, but man this is a deep serious stuff. I'm overwhelmed.
Your videos are always very informative and straight to the point, unlike other tutorials, where every character of code is analyzed on it's own. Keep up with these good videos
#pycharm
Loved it! Really really nice and descriptive video. Thanks for making this @mCoding.
You are the best in class!!! Salute!
I finally understood why multiple inheritance is dangerous. I didn't understood it in University. Thanks!
I love both PyCharm and CLion (the only editor I can not only stand for C++ but that I actually enjoy).
I seldom use super in my projects (personal and work), but this was a good explanation.
I looked for this A LOT through last week
You’re a mind reader and your content is amazing!
Thank you so much for helping the community to understand these concepts better! #pycharm
I started using super() due to the structure of a project (needed a parent class with some general methods and configs, and child classes made to inherit from it). I was exposed to it from a previous video from you, where you used to create a subclass dictionary to be able to call the subclass given the parameters ( this is something i ended up needing in said project). I already knew some of the stuff you expose here due to researching, but as always, you go the extra mile. Thanks for your content.
#pycharm
Very welcome and glad to hear you're doing your own experiments! That's the best way to learn it!
This was incredible. Thank you!
Side effect of this video: seeing a weird / operator in the __init__ of your custom Super implementation, looking it up and learning about positional only arguments. Loved this video!
Had to deal with a MRO nightmare I was building Custom Operators for an Apache Airflow application. Custom Operators basically inherit from other default Operators (to build custom solutions for your specific use-case), which in turn can inherit other operators and extra classes and so on until it gets to the BaseOperator class. To make matters worse, I wanted to merge 2 Operators into one.
Took me a long time to figure out what super() was actually calling and made me have to dig into a lot of stuff to find out. Your video is really straightforward and had it been released 1 year ago would have saved a couple of days of frustration. Awesome content as always! #pycharm
Can you explain why you needed custom operators as opposed to just Methods?
@@drygordspellweaver8761 I needed a simple way for the Data Analysts and Data Scientists to have their own self-serving Airflow instance. Since they already knew how to use an Operator, I built that way so I could "hide the wiring" behind that complicated process behind an interface in which was more familiar to them.
good Lord, so much depth inall of your videos. Keep up the great work
The more I watch this channel the more I know that I don't know Python at all.
Excellent video once again!
Wow, thanks!
Finally i understooood! Great explanation!!!
Thannks, You make difficult topics interesting and easy to learn
Hi, great video, you always do such in-depth research! Could you perhaps do a video on creating python bindings for C/C++? It can be rather useful but I have barely seen anyone discuss it seriously and thoroughly on youtube.
After so many years using python, I still learn a lot from this channel.
#Pycharm #CLion
Awesome advanced Python stuff. Already knew about how super works and the MRO but didn’t know about the implementation details. #pycharm
I just hope to be as insightful as this guy one day
It's always interesting to see in-depth explanations of how a language works under the hood, like this or the ",=" python operator you explained in a past video. Keep up the quality content :) #pycharm
That's an old one! You must be a long time viewer! Or a dedicated one at least!
Best videos on youtube dude, thank you so much
absolute knowledge.... , though most of these i knew olredy, but you are actually going gr8 sir.
how to get which class is being run by super , i did not know...thanks for that and really appreciable work sir, all hands down.
Always excited to see new videos from this channel! I appreciate seeing how things work behind the scenes. Thanks for the awesome content. #clion
Amazing video, thank you!
You just simply have to be one of the most informative channels on programming out there. #pycharm
Thank you very much I appreciate the praise
Hi James, I just started to seeing your videos, and they are of great quality. I'm a Mathematician and Data Scientist from Mexico. I just wanted to ask how did you achieve that deep knowledge of Python? Is there a set of subjects or books that you recommend? I have studied some basic OOP in Java. Thanks
Your videos always make me regret I ever got into Python. They're great!
Wow. Amazing explanation.
This was a bit over my head but so interesting and helpful. I love learning Python, and thank you for your videos. #pycharm
This video again shows how Python's compiler does magic things. And again emphasizes how it's very important to understand when things happen (in the compiler or in the interpreter).
Great video as always. You explained well the inner mechanics of the super() method. To understand super() perfectly, one must watch both Reymond's video and yours.
About the optional arguments: I used once super() inside a list comprehension. It took me quite a while to understand why Python threw an error each time. Apparently, list comprehensions (or generator expressions) are considered to be their own scope, and thus do not apply super() correctly. The remedy was to apply super() with two arguments; worked like a charm. #pycharm
Interesting and insightful as usual. What's really surprised me the part where you unraveled the mystery behind how zero argument super works, which also explains the older form of super. Thanks for your content! #clion
Always wanted too see info about super but never thought about it directly. Thanks! #pycharm
Oh this is perfect, i was just struggling with this like an hour ago.
I had some dataclass inheritance structure (frozen) and i had some classmethod as a factory which i wanted to inherit but it all fell apart.
I was just reading reading an article from Real Python about super() to clear some doubts and you posted a video about the same thing! #pycharm
The way the topics are choosen and presented is astonishing 🔥😇
#pycharm
Thanks for the video. quick question, could you share how did you remove your background from video and set computer screen as the background? which video recording software / tools you have used for making this video ? thanks,
I'm a data scientist/engineer/Web developer and I've made use of your videos in my work many times. Your channel is one of my training links I give out to new staff. Definitely would appreciate a Prof pycharm license for personal use #pycharm
Great video this concept, thanks
I really enjoy your deep dives into the nitty gritty details of python. I always come out feeling smarter.
#pycharm
thanks for clearing this out! Was confused about super until now #pycharm
It’s so interesting thank you!
Coming from Ruby's super, it's nice seeing how Python's works exactly for those more complex cases. #pycharm
There is one valid use case for the two and one argument form of super. I am a bit surprised you did not mention them. As you pointed out the first argument tells you where to look for the next in line in the MRO. If you pass in one of the parents explicitly you can ignore part of the MRO and look for methods or attributes starting from a fixed parent / sibling. Not that you need it all the time but it can be handy.
An excellent explanation as always! Personally I'm still a newbie when it comes to Python, I'm currently learning how to use classes in general, and this is the first time I'm hearing about class inheritance. Yet I could still follow a lot of the explanation, which says a lot about how good you are as a teacher! Thank you :)
#pycharm
Always getting a deeper understanding about Python through your videos.
Thank you for making such fantastic and no nonsense content!
Also #pycharm
Great tutorial! I've always wondered what super() did, especially with the syntax differences between 2.7 and 3, but this tutorial did a great job of explaining that! #clion
Great to hear! Also you get a CLion license! Contact me by email!
Thanks for the content...
Great video. This was one of those I was really waiting for.
Just a question.
So this all means that I can't correctly inherit from multiple parents, if those parents where not specifically built to be siblings, right?
That's some wonderful cursed knowledge right there, thank you.
Muchas gracias, enserió necesitaba mucho entender esto, y esta clase me ha ayudado bastante!
translate:
Thank you very much, I really needed to understand this, and this class has helped me a lot!.
very cool. even just the 99% bit - the rest was just gravy! thanks man!
I am using this frequently but this is far the best explanation .
#pycharm
Time to learn something I never needed to know about Python at this level of depth. #pycharm
It's always great to cement up knowledge gained from documentation through your videos. Thanks! #pycharm
Awesome explaination. I'm revising Python after so long and these type of videos helps us to get a deeper knowledge of certain aspect of the code.
#pycharm