In this case it was the same as a BFS, but in general it does not have to be! See the L(Z) example in en.wikipedia.org/wiki/C3_linearization for a case where it is not a BFS.
Technically, it does a depth-first, left-right search and can contain duplicates. Then, it removes all but one of the duplicates. The last duplicate is preserved, all else are removed. Eg: class A: pass class B(A): pass class C(A): pass class D(B, C): pass so depth-first, left-right search list for 'D' would be something like this: ['D', 'B', 'A', 'object', 'C', 'A', 'object'] Now, all earlier duplicates will be removed for the __mro__, i.e the first 'A', and the first 'object' will be removed from list. ['D', 'B', 'A'(first 'A', hence removed), 'object'(first 'object', hence removed), 'C', 'A', 'object'] the final list is this: ['D', 'B', 'C', 'A', 'object']
i spent days treading in convoluted documentation to finally understand MRO, multiple inheritance, and super (had to also get through descriptors to understand the bounding mechanics)... and here it is, a 20 minute video that explains everything. this channel is way too underrated.
Thanks for the kind words! And I also recommend "Super is considered super"! It's linked and Hettinger was (I believe) the one who coined the "next in line" phrase.
@@mCoding If anything, it's worth watching for the volunteer part (explaining cooperative inheritance and MRO) at very least. Visuals like that make things a lot easier to comprehend/memorize. I'm not a fan of some of Raymond's teaching techniques but this specific part was brilliant.
With that experience maybe you could make the documentation better? I’m not trying to be one of the “then open a pull request!” guys but it sounds like you have special insight that might help future users.
The *level* of this channel is 👌. I've been coding in Python for quite awhile and I always learn stuff here. It's really refreshing when a lot of what is out there is like "how to use dicts in Python!"
So true, it's high level, but it doesn't use an overwhelming amount of jargon. Learning a new programming language is a pain, because 90% of the material and tutorials are "This is the syntax for a for loop", while the remaining 10% is an argument on stack exchange about the best boosting the efficiency of a highly specific package from 1994 with some compiler meta-optimization. It's either teaching a complete beginner how to code for the first time, or a set of code-words that will remind an expert of how to do something. When I almost always want is help on turning my ok-ish code, into good code. I wish he did material on, say, C# or Julia, so I could go from a basic understanding of the surface feature of the language to having an idea of how the structure of it works.
Thanks for being so clean and straight-to-the-point in how you present your topics. Your videos are quite like 3Blue1Brown but for programming. Keep up the dry humor, stellar stuff. #pycharm
This is why multiple inheritance, especially "diamond" multiple inheritance is evil. Even if you're one of the 0.01% who knows how it works, it will still trip you up when you dont expect it (that is, 99% of the time). Having recently come across C#'s solution to this (a class can only inherit from 1 class, but can inherit from as many Interfaces as it wants), I now think python's approach is too liberal. Of course that's the whole python ethos, but sometimes it creates far more traps than necessary. #pycharm
Here the trap isn't that problematic imo, once you inherit from multiple classes there is an obvious problem with super. Best practise I think is to specify which parent function you want to call
So you've discovered interfaces. My favorite C# feature. Initially they seem hard but once you get the hang of it you'll never go back to MI. When you get a bit deeper you'll hit covariance / contravariance with interfaces, and the slightly ugly type parameter constraints that all kind of tie together. C#'s type system is not quite perfection.
James: your expert knowledge of not just Python, but C, and CPython implementation - along with dense and info-rich presentation - make for a quite rare resource of deep, under-the-hood understanding of Python. Thanks for putting in the work!
Even though I am aware about using super and cooperative multiple inheritance when I watched the iconic talk of Mr. Raymond Hettinger on super()... But even then, my mind was simply in awe when you dived into the details... I never even imagined that super class is sort of implementing a Proxy pattern...
The only time I used multiple inheritance it was a big mistake, it became impossible to have the classes cooperate nicely. A slightly more verbose solution based on composition solved the same problem without all the subtle bugs #pycharm
@@mCoding Could you please make a video where you show with a few different examples to replace a solution with inheritance with functional composition?
That's what always happens with multiple inheritance in python. It's a magical feature: the instant you implement it, the code becomes unmaintainable. Not at some future date, not after some drift, etc, no: the very next thing you try to change will be extremely difficult. It's truly impressive how bad it is.
Another great video, the first half was satisfyingly confirming some practices I have picked up by using python for a while, and the latter half was part mind-boggling, part "oh please don't do that". Learned a lot as always, thank you James! #pycharm
Very nice! Your example code at the end is the part that was missing in PEP 3135 back when it was written, it would have cleared up how super works for most people. For a good example, look at PEP 343 - with Statement, where the "Specification" section shows exactly how it works in a very clear fashion.
In current example it is the right form of using dunder getattr method. Since you don't want to get the attribute of an instance of this class but the self.obj, that will make the code work excatly as it should. Changing dunder getattr to getattribute will cause the RecusrionError because it will invoke itself infenetily long, as you're trying to get the attribute not of self.obj but of self - an instance of current class.
A good addition for cooperative inheritance is that the root class' sink method verifies that both *args and **kwargs are empty at that point and throws an error if either isn't. This helps prevent unnecessary arguments being passed somewhere in the chain that stick around after changes/refactoring as the *args and **kwargs in the signatures of children mean that static code analysis often won't see anything wrong.
I remember that I watched Raymond Hettinger's "Super considered super!", understood a little more about the MRO but could not understand how `super()` works. Now I have watched this video and I have learnt something new today! Thank you! I still don't fully understand the `super(...)` with argument(s) form, but I guess/hope I don't need it since I don't need to consider my job security. 😂
IIRC, you can rebind super to have the compiler magic keep working while also allowing you to change its behaviour (it's definitely the case that using super bound to another name causes the compiler magic to fail) EDIT: To be clear, if you had written `class super:` instead of `class Super:` it would be a drop-in replacement
It's always interesting to see in-depth explanations of how a language works under the hood, like this or the ",=" python operator you explained in a past video. Keep up the quality content :) #pycharm
Finally, a video that covers the inner details of super and how it works, also the pure python implementation of super is awesome, learnt a lot from this video, thank you so much Murphy. #pycharm
@@josephlyons3393 At least we don't have a whiny compiler. :) Just kidding, I like the rust's approach. Cppckeck for cpp is sometimes all what you need tho.
@@embeddor3023Rustc certainly is soul crushing, lol. C++ was my main language for many years, but I mostly dev in either Rust or Python these days, depending on the task.
To be fair, it's python, which is about convenience over speed. If you want method and property lookups to be fast, you should probably be using a different kind of language
@@berylliosis5250 Considering that attribute/method lookups are one of the most common things you do in a programming language, it's quite frightening that what takes one machine instruction in a compiled language takes upwards of 100 *bytecode* instructions, including both branches and memory fetches. I hope (and think, but am too lazy now to check) that Python does some optimizations here to bypass this cost because I've never really felt that Python is that slow in comparison. I do expect a certain level of speed from any language.
I would love to see an explanation of the Walrus (:=) operator. How it works,tips and tricks, and use case scenarios. Mostly I've missunderstood when i should/can use it. My code often feels non-reable when using it. Loved this video (and ofc your previous ones)! #pycharm
I find it quite enlightening when you dive deeper into how these features work rather than working with the superficial layer of when to use/ how to use these features. I always come away with some useful knowledge I can apply to my code. #pycharm
Well, tbh. in most cases it's not a big deal because you can compose functions pretty easily by explicitly calling them. You don't need "object oriented" programming (more like inheritance) in these situations.
It seems like it, but it's actually far less complicated _in practice_ than in many other languages, like C++ (public vs private inheritance, recursive inheritance [CRTP], pure abstract classes, constness concerns, public/protected/private members, etc. -- and that's before we add templating!). In both cases, though, you should really avoid inheritance for code reuse, anyway. Inheritance should be used for polymorphism. Only in some rare cases when following certain design patterns (or idioms) should you inherit anything but the interface, and in those cases, you should just follow the pattern. Generic programming is usually closer to what you want with code reuse; Python has this with duck typing, while languages like C++ do this with templates, and some others like Java fake it (somewhat poorly) with type erasure.
Had to deal with a MRO nightmare I was building Custom Operators for an Apache Airflow application. Custom Operators basically inherit from other default Operators (to build custom solutions for your specific use-case), which in turn can inherit other operators and extra classes and so on until it gets to the BaseOperator class. To make matters worse, I wanted to merge 2 Operators into one. Took me a long time to figure out what super() was actually calling and made me have to dig into a lot of stuff to find out. Your video is really straightforward and had it been released 1 year ago would have saved a couple of days of frustration. Awesome content as always! #pycharm
@@drygordspellweaver8761 I needed a simple way for the Data Analysts and Data Scientists to have their own self-serving Airflow instance. Since they already knew how to use an Operator, I built that way so I could "hide the wiring" behind that complicated process behind an interface in which was more familiar to them.
Note to self: never use inheritance in Python if I want my code to run fast. Seeing what goes on behind the scenes is amazing, and a bit scary... the VM must be doing *so* much work every time a super() gets called XD
after like 8 years of programming I've come to the conclusion to use inheritance as little as possible it just creates a mess in places you never think of
It is impressive, how Python implements all these functionality _with Python_ - from a technological point of view. Python, as an interpreter language, implements itself by using Python... fascinating! But from the perspective of a user of the language ... all this black magic everywhere ... meh. :/ In other languages, using "clever coding" is regarded as something bad ...
I'm assuming that super is implemented in C (at least for cpython) and not in pure python since it's a built-in. But it could be implemented in python, aside from the class injection hack.
Compared to most other languages, python has very little black magic. Other languages also have things like implicit `this` variable, which feels more like black magic to me
I think this is a case of "practicality beats purity" :D I remember programming in Python 2, which didn't have this black magic. This meant that every. single. super call needed to be the two-argument form... Believe me, it gets old very quickly, apart from being error-prone, as you now have to mention the class you're in everywhere.
I started using super() due to the structure of a project (needed a parent class with some general methods and configs, and child classes made to inherit from it). I was exposed to it from a previous video from you, where you used to create a subclass dictionary to be able to call the subclass given the parameters ( this is something i ended up needing in said project). I already knew some of the stuff you expose here due to researching, but as always, you go the extra mile. Thanks for your content. #pycharm
Multiple inheritance needs a carefully designed class hierarchy. As much as possible, structure your Python classes to emulate the functionality of the single inheritance + interfaces/mix-ins model. In particular, only the principal parent class of a class should descend from anything other than object, thus mitigating the diamond problem. But also consider composition may be a better option for secondary parents.
Side effect of this video: seeing a weird / operator in the __init__ of your custom Super implementation, looking it up and learning about positional only arguments. Loved this video!
I looked for this A LOT through last week You’re a mind reader and your content is amazing! Thank you so much for helping the community to understand these concepts better! #pycharm
Your videos are always very informative and straight to the point, unlike other tutorials, where every character of code is analyzed on it's own. Keep up with these good videos #pycharm
I am glad I came across your channel. Just realised I've been more focused on the 'how' things work in python but didn't give much thought to the 'why'...good job mate. #pycharm
So in simple terms: Python is mixing up terminology and making it more confusing than it needs to be. In basically all other languages "super" means "call the parent" but in python it is "call whatever strange thing we find in the inheritance-tree next that does happen to have that". Instead of the language providing consistent behaviour it is up to the developer to structure the inheritance in a way that just "happens to work" with super..... And then we have the thing that it is Python - a "language" with no real standard as it is constantly changing and breaking fundamental behaviour of the code - like the language also changed the definition of the MRO. (Worse yet - you can manipulate the MRO). Well - that is the thing that makes python so useful for clobbering together something to "just do this", but it is also the same thing that makes it nigh impossible to use for large projects.
This video again shows how Python's compiler does magic things. And again emphasizes how it's very important to understand when things happen (in the compiler or in the interpreter).
I love both PyCharm and CLion (the only editor I can not only stand for C++ but that I actually enjoy). I seldom use super in my projects (personal and work), but this was a good explanation.
I'm a data scientist/engineer/Web developer and I've made use of your videos in my work many times. Your channel is one of my training links I give out to new staff. Definitely would appreciate a Prof pycharm license for personal use #pycharm
There is one valid use case for the two and one argument form of super. I am a bit surprised you did not mention them. As you pointed out the first argument tells you where to look for the next in line in the MRO. If you pass in one of the parents explicitly you can ignore part of the MRO and look for methods or attributes starting from a fixed parent / sibling. Not that you need it all the time but it can be handy.
Those theoretical questions were bait-and-switch tricks. It would have been less confusing if you had shown all the information before asking the question rather than hiding the most important part that makes it possible to accurately assess the question.
Personally, for those very few times it _appears_ ambiguous which defining class the `super` will go to, I either use the super call with arguments, or just outright call `superclass_name.function_name(self)` explicitly. That way I don't have to require the reader to untangle the MRO algorithm. The downside of course is that now the super classes have to be named multiple places. The class's definition and any place I need to use this trick. But I consider that worth not requiring knowledge of the "deep magics" of Python
absolute knowledge.... , though most of these i knew olredy, but you are actually going gr8 sir. how to get which class is being run by super , i did not know...thanks for that and really appreciable work sir, all hands down.
the comments at 3:00 seems wrong, it seems that it does go to A's f but A's f IS Root's f. So it does go to A's f in that case, which is the same f as Root's f.
hey, i just came up to that video an i really love it!! although, i think super and mro are far from the best python feature, and i made an alternative to super for fun. I think it's weird to assume it's possible to order multiple inheritence trees, it's weird the target of super is so implicit, and depends on context, it's weird that child class affect the super behavior in its parent, it feels like a side effect, and it's weird that mro can fail, and makes some class definition impossible. the alternative i produced for super (i called it __as_parent__) follow those rules : 1) it should reliably target the parent class it targets, no matter what other child class you decide to create later on 2) in case of multiple inheritence, it should be passed the parent it targets as argument (you can then make one call per parent method you wanna refer) 3) it should be able to stay implicit when it doesn't need to be explicit, no need to specify the parent when there's only one 4) it should be able to target ancestors, direct or not (it's needed to make it work with the alternative to mro i produce, in some weird cases) the alternative i produced for mro (i call it explicit method resolution) follow those rules : 1) any method / attribute present in the class definition should be the one resolved to (other rules assume the method / attribute is no present in the class definition) 2) in case a method / attribute is not present in the class definition and in any ancestors definition, it should raise an AttributeError 3) in case only one parent can resolve it, it should be the one resolved to 4) in case multiple parents can resolve it, it should raise an ExplicitResolutionRequired error (i'll explain how to solve it later) 5) in case one parent raises an ExplicitResolutionResquired error, this error is transmitted to the child 6) in case multiple parent can source it from on single source (diamond shaped inheritence), the ExplicitResolutionRequired error is not needed To solve an ExplicitResolutionRequired error, you can just define the method / attribute in the child class definition. (method can then refer to parents method through the use of super or my __as_parent__ alternative) Do you think it make sense? here's the github repository : github.com/malmiteria/super-alternative-to-super
I love the idea and huge props for implementing it! That's the blessing and the curse of such a dynamic language. Don't like something? You can completely write your own.
I predicted that super uses __locals__ and type() but the Proxy concept was the missing piece. Very enlightening. I learned about proxy objects last yearish but haven't intentionally used the pattern in my own software yet, so it doesn't come to mind immediately as a solution. Gotta find an excuse to apply the concept in my own code sometime so the idea sticks
Yeah proxy objects are one of those things that make a lot of python magic happen. They can be useful, but they can also be confusing for users sometimes.
10:05 I think there's a potential problem here with using a mix of explicit and packed arguments. The explicit arguments will be absent from the packed arguments, so then would also be absent from the super call. Eg if the next sibling class also needs the argument `validators`, it won't get it because here `kwargs` doesn't contain `validators`. If, however, you preserve all the arguments in the super call, then there may instead be a problem if the base class doesn't accept the extra arguments.
Nice video, though you didn't fully specify the MRO search algorithm (C3 linearization). I fully agree with Mark Lutz (Learning Python) on the concepts of "Cooperative Inheritance": best not to do it, since any programmer using a library you write using cooperative inheritance can break your code by simply adding a new mis-behaving class in the wrong level in your hierarchy.
I did use it once, it frelled up my code... but what it did for me was to prove that I should not have used classes for that problem in the first place. The need for "super" is a super-sized warning flag that the design is absolutely flawed.
Hi James, I just started to seeing your videos, and they are of great quality. I'm a Mathematician and Data Scientist from Mexico. I just wanted to ask how did you achieve that deep knowledge of Python? Is there a set of subjects or books that you recommend? I have studied some basic OOP in Java. Thanks
Hi, great video, you always do such in-depth research! Could you perhaps do a video on creating python bindings for C/C++? It can be rather useful but I have barely seen anyone discuss it seriously and thoroughly on youtube.
This channel has really helped me level up my Python skills. Videos like this are really helpful for people like me who want an “under the hood” understanding of the language. #pycharm
The important lesson here is that when MRO starts knocking on your door, it's best to just refactor your code to not use convoluted inheritance trees. Multiple inheritance is a mistake, and while I don't berate Python for supporting the use-case, I have yet to encounter a problem where the code couldn't be refactored to be more clear, without multiple inheritance.
16:10 This might sound nitpicky, but I think the wording you used here ("contains the class that is currently executing") is a bit incomplete and could mislead people who don't know better into thinking that using the ___class___ _attribute_ does the same thing. I would rather define what this ___class___ variable contains as "the class that the current function is defined under", or something similar.
In my opinion, if you need to think about MRO, then it's time to refactor your code. I use inheritance only when I create interface. Composition is the answer most of the time...
I noticed that you have a bit of background noise on your microphone. Have you tried out using noise suppression, such as that provided by OBS's built-in Noise Suppression filter, or Krisp's free desktop software?
Yes I use obs's built-in noise filter. I haven't changed my setup in a long time but many people are saying there is noise now. I double checked everything is turned on and setup as usual but I haven't diagnosed the issue yet.
![[UA] Eternal Fire проти NAVI | EPL Season 20 Malta](http://i.ytimg.com/vi/chvlsDygrZM/mqdefault.jpg)
Isn't the search that the MRO is doing the breadth-first search?
In this case it was the same as a BFS, but in general it does not have to be! See the L(Z) example in en.wikipedia.org/wiki/C3_linearization for a case where it is not a BFS.
@@mCoding my assumptions keep breaking and I love it.
Technically, it does a depth-first, left-right search and can contain duplicates. Then, it removes all but one of the duplicates. The last duplicate is preserved, all else are removed.
Eg:
class A: pass
class B(A): pass
class C(A): pass
class D(B, C): pass
so depth-first, left-right search list for 'D' would be something like this:
['D', 'B', 'A', 'object', 'C', 'A', 'object']
Now, all earlier duplicates will be removed for the __mro__, i.e the first 'A', and the first 'object' will be removed from list.
['D', 'B', 'A'(first 'A', hence removed), 'object'(first 'object', hence removed), 'C', 'A', 'object']
the final list is this:
['D', 'B', 'C', 'A', 'object']
@@mCodingthat algorithm looks like an O(n^3) mess lol
i spent days treading in convoluted documentation to finally understand MRO, multiple inheritance, and super (had to also get through descriptors to understand the bounding mechanics)... and here it is, a 20 minute video that explains everything. this channel is way too underrated.
Raymond Hettinger's old talk "Super is considered super" is fairly good, too
Thanks for the kind words! And I also recommend "Super is considered super"! It's linked and Hettinger was (I believe) the one who coined the "next in line" phrase.
@@mCoding If anything, it's worth watching for the volunteer part (explaining cooperative inheritance and MRO) at very least. Visuals like that make things a lot easier to comprehend/memorize. I'm not a fan of some of Raymond's teaching techniques but this specific part was brilliant.
With that experience maybe you could make the documentation better? I’m not trying to be one of the “then open a pull request!” guys but it sounds like you have special insight that might help future users.
I've been programming in Python professionally for almost 10 years and I learned something in this video.
Glad to hear!
Same!
does that mean these things are not going to be used in industry?
@@TemporaryForstudy not if you want to keep working in that industry ;-)
@@Ratimus_ Thanks for the information. Are you a python developer?
The *level* of this channel is 👌. I've been coding in Python for quite awhile and I always learn stuff here. It's really refreshing when a lot of what is out there is like "how to use dicts in Python!"
yeah that the problem, he is teaching super , to advanced people.
So true, it's high level, but it doesn't use an overwhelming amount of jargon. Learning a new programming language is a pain, because 90% of the material and tutorials are "This is the syntax for a for loop", while the remaining 10% is an argument on stack exchange about the best boosting the efficiency of a highly specific package from 1994 with some compiler meta-optimization. It's either teaching a complete beginner how to code for the first time, or a set of code-words that will remind an expert of how to do something. When I almost always want is help on turning my ok-ish code, into good code.
I wish he did material on, say, C# or Julia, so I could go from a basic understanding of the surface feature of the language to having an idea of how the structure of it works.
Exactly! And it's always some of the most error-prone structures that you can't understand!
Love your videos!
Thanks for being so clean and straight-to-the-point in how you present your topics. Your videos are quite like 3Blue1Brown but for programming. Keep up the dry humor, stellar stuff. #pycharm
This is why multiple inheritance, especially "diamond" multiple inheritance is evil. Even if you're one of the 0.01% who knows how it works, it will still trip you up when you dont expect it (that is, 99% of the time). Having recently come across C#'s solution to this (a class can only inherit from 1 class, but can inherit from as many Interfaces as it wants), I now think python's approach is too liberal. Of course that's the whole python ethos, but sometimes it creates far more traps than necessary. #pycharm
Too many people take my job security comment a bit too seriously!
Here the trap isn't that problematic imo, once you inherit from multiple classes there is an obvious problem with super.
Best practise I think is to specify which parent function you want to call
So you've discovered interfaces. My favorite C# feature. Initially they seem hard but once you get the hang of it you'll never go back to MI. When you get a bit deeper you'll hit covariance / contravariance with interfaces, and the slightly ugly type parameter constraints that all kind of tie together. C#'s type system is not quite perfection.
That's precisely the case with Java as well.
Don't all classes derive from object, making any multiple inheritance diamond shaped?
James: your expert knowledge of not just Python, but C, and CPython implementation - along with dense and info-rich presentation - make for a quite rare resource of deep, under-the-hood understanding of Python. Thanks for putting in the work!
Amazing insights into the super() function implementation in Python! Thank you for your great work, I always learn something new!
Glad it was helpful!
Even though I am aware about using super and cooperative multiple inheritance when I watched the iconic talk of Mr. Raymond Hettinger on super()...
But even then, my mind was simply in awe when you dived into the details... I never even imagined that super class is sort of implementing a Proxy pattern...
The only time I used multiple inheritance it was a big mistake, it became impossible to have the classes cooperate nicely. A slightly more verbose solution based on composition solved the same problem without all the subtle bugs
#pycharm
As is always a question you should ask, just because you *can* solve the problem with multiple inheritance, *should* you?
@@mCoding I believe 99,99999% of the time the ansewer is sounding NO.
Rule 2 of OOP: Favor Object composition over inheritance
@@mCoding Could you please make a video where you show with a few different examples to replace a solution with inheritance with functional composition?
That's what always happens with multiple inheritance in python. It's a magical feature: the instant you implement it, the code becomes unmaintainable. Not at some future date, not after some drift, etc, no: the very next thing you try to change will be extremely difficult. It's truly impressive how bad it is.
The 2 argument super used to be the required way of calling super in python2. So glad we don't need that anymore
May Python 2 rest in peace, forever!
Another great video, the first half was satisfyingly confirming some practices I have picked up by using python for a while, and the latter half was part mind-boggling, part "oh please don't do that". Learned a lot as always, thank you James! #pycharm
Glad you enjoyed it!
Very nice! Your example code at the end is the part that was missing in PEP 3135 back when it was written, it would have cleared up how super works for most people. For a good example, look at PEP 343 - with Statement, where the "Specification" section shows exactly how it works in a very clear fashion.
13:18 little mistake. __getattr__ is called whenever getting attribute is failed, __getattribute__ is called everytime you trying to get an attribute.
In current example it is the right form of using dunder getattr method. Since you don't want to get the attribute of an instance of this class but the self.obj, that will make the code work excatly as it should. Changing dunder getattr to getattribute will cause the RecusrionError because it will invoke itself infenetily long, as you're trying to get the attribute not of self.obj but of self - an instance of current class.
A good addition for cooperative inheritance is that the root class' sink method verifies that both *args and **kwargs are empty at that point and throws an error if either isn't. This helps prevent unnecessary arguments being passed somewhere in the chain that stick around after changes/refactoring as the *args and **kwargs in the signatures of children mean that static code analysis often won't see anything wrong.
Fantastic introduction to super() and MRO, thanks.
My pleasure! Have a super day!
I remember that I watched Raymond Hettinger's "Super considered super!", understood a little more about the MRO but could not understand how `super()` works.
Now I have watched this video and I have learnt something new today! Thank you!
I still don't fully understand the `super(...)` with argument(s) form, but I guess/hope I don't need it since I don't need to consider my job security. 😂
IIRC, you can rebind super to have the compiler magic keep working while also allowing you to change its behaviour (it's definitely the case that using super bound to another name causes the compiler magic to fail)
EDIT: To be clear, if you had written `class super:` instead of `class Super:` it would be a drop-in replacement
This was the most I have learned about python in any 20 min ever.
Subscribed
It's always interesting to see in-depth explanations of how a language works under the hood, like this or the ",=" python operator you explained in a past video. Keep up the quality content :) #pycharm
That's an old one! You must be a long time viewer! Or a dedicated one at least!
Finally, a video that covers the inner details of super and how it works, also the pure python implementation of super is awesome, learnt a lot from this video, thank you so much Murphy. #pycharm
Very welcome and glad you enjoyed!
You are really good at teaching. I like how you build up the complexity. Nice!
As a C/C++ dev, all this runtime stuff python is doing just to do basic static things is horrifying.
As a Rust dev, all this unsafe UB in C/C++ is horrifying! :D
@@josephlyons3393 At least we don't have a whiny compiler. :)
Just kidding, I like the rust's approach. Cppckeck for cpp is sometimes all what you need tho.
@@embeddor3023Rustc certainly is soul crushing, lol. C++ was my main language for many years, but I mostly dev in either Rust or Python these days, depending on the task.
And here I thought “How complicated can super() really be?” It’s a joy to learn from you. Keep up the great content! #pycharm
It's a bit scary how much work at runtime goes behind a simple attribute/method lookup.
To be fair, it's python, which is about convenience over speed. If you want method and property lookups to be fast, you should probably be using a different kind of language
@@berylliosis5250 Considering that attribute/method lookups are one of the most common things you do in a programming language, it's quite frightening that what takes one machine instruction in a compiled language takes upwards of 100 *bytecode* instructions, including both branches and memory fetches.
I hope (and think, but am too lazy now to check) that Python does some optimizations here to bypass this cost because I've never really felt that Python is that slow in comparison. I do expect a certain level of speed from any language.
Love the 1-letter variables in your functions. Right after coming from one of your "nooby habits" videos too.
Intermediate programming channel are so rare.
It's always a pleasure watching your videos !
#clion
I have been wondering whether a function like this exists for a long time. Thank you!
Very welcome!
I would love to see an explanation of the Walrus (:=) operator. How it works,tips and tricks, and use case scenarios. Mostly I've missunderstood when i should/can use it. My code often feels non-reable when using it.
Loved this video (and ofc your previous ones)!
#pycharm
I find it quite enlightening when you dive deeper into how these features work rather than working with the superficial layer of when to use/ how to use these features. I always come away with some useful knowledge I can apply to my code. #pycharm
Object oriented programming in Python is complicated. Thank you for this video!
Well, tbh. in most cases it's not a big deal because you can compose functions pretty easily by explicitly calling them. You don't need "object oriented" programming (more like inheritance) in these situations.
It seems like it, but it's actually far less complicated _in practice_ than in many other languages, like C++ (public vs private inheritance, recursive inheritance [CRTP], pure abstract classes, constness concerns, public/protected/private members, etc. -- and that's before we add templating!).
In both cases, though, you should really avoid inheritance for code reuse, anyway. Inheritance should be used for polymorphism. Only in some rare cases when following certain design patterns (or idioms) should you inherit anything but the interface, and in those cases, you should just follow the pattern. Generic programming is usually closer to what you want with code reuse; Python has this with duck typing, while languages like C++ do this with templates, and some others like Java fake it (somewhat poorly) with type erasure.
Your python knowledge is truly frightening... I like the video, but man this is a deep serious stuff. I'm overwhelmed.
Wanted to get into open source dev. One of the main projects I wanted to get into is pretty #clion oriented would love a key!
Had to deal with a MRO nightmare I was building Custom Operators for an Apache Airflow application. Custom Operators basically inherit from other default Operators (to build custom solutions for your specific use-case), which in turn can inherit other operators and extra classes and so on until it gets to the BaseOperator class. To make matters worse, I wanted to merge 2 Operators into one.
Took me a long time to figure out what super() was actually calling and made me have to dig into a lot of stuff to find out. Your video is really straightforward and had it been released 1 year ago would have saved a couple of days of frustration. Awesome content as always! #pycharm
Can you explain why you needed custom operators as opposed to just Methods?
@@drygordspellweaver8761 I needed a simple way for the Data Analysts and Data Scientists to have their own self-serving Airflow instance. Since they already knew how to use an Operator, I built that way so I could "hide the wiring" behind that complicated process behind an interface in which was more familiar to them.
Note to self: never use inheritance in Python if I want my code to run fast. Seeing what goes on behind the scenes is amazing, and a bit scary... the VM must be doing *so* much work every time a super() gets called XD
after like 8 years of programming I've come to the conclusion to use inheritance as little as possible it just creates a mess in places you never think of
I love the level of depth you manage to go into in such a short time. Few guys on UA-cam deliver this stuff as well as you do. Also #pycharm :)
Discord gang 🤙🤙
Always appreciated!
I love these metaprogramming videos. The best way to explain a complex language features is to show the implementation.
It is impressive, how Python implements all these functionality _with Python_ - from a technological point of view. Python, as an interpreter language, implements itself by using Python... fascinating!
But from the perspective of a user of the language ... all this black magic everywhere ... meh. :/ In other languages, using "clever coding" is regarded as something bad ...
I'm assuming that super is implemented in C (at least for cpython) and not in pure python since it's a built-in. But it could be implemented in python, aside from the class injection hack.
Compared to most other languages, python has very little black magic. Other languages also have things like implicit `this` variable, which feels more like black magic to me
I think this is a case of "practicality beats purity" :D
I remember programming in Python 2, which didn't have this black magic. This meant that every. single. super call needed to be the two-argument form... Believe me, it gets old very quickly, apart from being error-prone, as you now have to mention the class you're in everywhere.
Awesome explanation, especially the part on how super works underneath the hood!! Thanks a lot.
Loved it! Really really nice and descriptive video. Thanks for making this @mCoding.
You are the best in class!!! Salute!
Wow. You put in a LOT of detective work on this. Thanks.
I finally understood why multiple inheritance is dangerous. I didn't understood it in University. Thanks!
Wow - that was complicated but enlightening. Your videos are great!
I've been asked many times about a clear explanation of super(), and here it is!
I started using super() due to the structure of a project (needed a parent class with some general methods and configs, and child classes made to inherit from it). I was exposed to it from a previous video from you, where you used to create a subclass dictionary to be able to call the subclass given the parameters ( this is something i ended up needing in said project). I already knew some of the stuff you expose here due to researching, but as always, you go the extra mile. Thanks for your content.
#pycharm
Very welcome and glad to hear you're doing your own experiments! That's the best way to learn it!
After so many years using python, I still learn a lot from this channel.
#Pycharm #CLion
This channel is pure gem!
Multiple inheritance needs a carefully designed class hierarchy. As much as possible, structure your Python classes to emulate the functionality of the single inheritance + interfaces/mix-ins model. In particular, only the principal parent class of a class should descend from anything other than object, thus mitigating the diamond problem. But also consider composition may be a better option for secondary parents.
Side effect of this video: seeing a weird / operator in the __init__ of your custom Super implementation, looking it up and learning about positional only arguments. Loved this video!
Awesome advanced Python stuff. Already knew about how super works and the MRO but didn’t know about the implementation details. #pycharm
I looked for this A LOT through last week
You’re a mind reader and your content is amazing!
Thank you so much for helping the community to understand these concepts better! #pycharm
The way the topics are choosen and presented is astonishing 🔥😇
#pycharm
You just simply have to be one of the most informative channels on programming out there. #pycharm
Thank you very much I appreciate the praise
The more I watch this channel the more I know that I don't know Python at all.
Excellent video once again!
Wow, thanks!
Oh my God thank you. Having a video this concise and clear is really really helpful.
Using super with the two arguments to call any object's function is blowing my mind!
I just hope to be as insightful as this guy one day
Your videos are always very informative and straight to the point, unlike other tutorials, where every character of code is analyzed on it's own. Keep up with these good videos
#pycharm
Coming from Ruby's super, it's nice seeing how Python's works exactly for those more complex cases. #pycharm
This is actually insane, the cursed local var situation is just shocking.
I am glad I came across your channel. Just realised I've been more focused on the 'how' things work in python but didn't give much thought to the 'why'...good job mate. #pycharm
good Lord, so much depth inall of your videos. Keep up the great work
Amazing. Your videos are so consistently great. Good work!
So in simple terms:
Python is mixing up terminology and making it more confusing than it needs to be. In basically all other languages "super" means "call the parent" but in python it is "call whatever strange thing we find in the inheritance-tree next that does happen to have that". Instead of the language providing consistent behaviour it is up to the developer to structure the inheritance in a way that just "happens to work" with super.....
And then we have the thing that it is Python - a "language" with no real standard as it is constantly changing and breaking fundamental behaviour of the code - like the language also changed the definition of the MRO. (Worse yet - you can manipulate the MRO).
Well - that is the thing that makes python so useful for clobbering together something to "just do this", but it is also the same thing that makes it nigh impossible to use for large projects.
This video again shows how Python's compiler does magic things. And again emphasizes how it's very important to understand when things happen (in the compiler or in the interpreter).
I love both PyCharm and CLion (the only editor I can not only stand for C++ but that I actually enjoy).
I seldom use super in my projects (personal and work), but this was a good explanation.
#PyCharm Could have never have guessed that MRO was being used in resolving super calls.
I'm a data scientist/engineer/Web developer and I've made use of your videos in my work many times. Your channel is one of my training links I give out to new staff. Definitely would appreciate a Prof pycharm license for personal use #pycharm
Your videos always make me regret I ever got into Python. They're great!
There is one valid use case for the two and one argument form of super. I am a bit surprised you did not mention them. As you pointed out the first argument tells you where to look for the next in line in the MRO. If you pass in one of the parents explicitly you can ignore part of the MRO and look for methods or attributes starting from a fixed parent / sibling. Not that you need it all the time but it can be handy.
Those theoretical questions were bait-and-switch tricks. It would have been less confusing if you had shown all the information before asking the question rather than hiding the most important part that makes it possible to accurately assess the question.
Personally, for those very few times it _appears_ ambiguous which defining class the `super` will go to, I either use the super call with arguments, or just outright call `superclass_name.function_name(self)` explicitly.
That way I don't have to require the reader to untangle the MRO algorithm.
The downside of course is that now the super classes have to be named multiple places. The class's definition and any place I need to use this trick.
But I consider that worth not requiring knowledge of the "deep magics" of Python
absolute knowledge.... , though most of these i knew olredy, but you are actually going gr8 sir.
how to get which class is being run by super , i did not know...thanks for that and really appreciable work sir, all hands down.
the comments at 3:00 seems wrong, it seems that it does go to A's f but A's f IS Root's f.
So it does go to A's f in that case, which is the same f as Root's f.
hey, i just came up to that video an i really love it!!
although, i think super and mro are far from the best python feature, and i made an alternative to super for fun.
I think it's weird to assume it's possible to order multiple inheritence trees, it's weird the target of super is so implicit, and depends on context, it's weird that child class affect the super behavior in its parent, it feels like a side effect, and it's weird that mro can fail, and makes some class definition impossible.
the alternative i produced for super (i called it __as_parent__) follow those rules :
1) it should reliably target the parent class it targets, no matter what other child class you decide to create later on
2) in case of multiple inheritence, it should be passed the parent it targets as argument (you can then make one call per parent method you wanna refer)
3) it should be able to stay implicit when it doesn't need to be explicit, no need to specify the parent when there's only one
4) it should be able to target ancestors, direct or not (it's needed to make it work with the alternative to mro i produce, in some weird cases)
the alternative i produced for mro (i call it explicit method resolution) follow those rules :
1) any method / attribute present in the class definition should be the one resolved to (other rules assume the method / attribute is no present in the class definition)
2) in case a method / attribute is not present in the class definition and in any ancestors definition, it should raise an AttributeError
3) in case only one parent can resolve it, it should be the one resolved to
4) in case multiple parents can resolve it, it should raise an ExplicitResolutionRequired error (i'll explain how to solve it later)
5) in case one parent raises an ExplicitResolutionResquired error, this error is transmitted to the child
6) in case multiple parent can source it from on single source (diamond shaped inheritence), the ExplicitResolutionRequired error is not needed
To solve an ExplicitResolutionRequired error, you can just define the method / attribute in the child class definition. (method can then refer to parents method through the use of super or my __as_parent__ alternative)
Do you think it make sense?
here's the github repository : github.com/malmiteria/super-alternative-to-super
I love the idea and huge props for implementing it! That's the blessing and the curse of such a dynamic language. Don't like something? You can completely write your own.
I predicted that super uses __locals__ and type() but the Proxy concept was the missing piece. Very enlightening. I learned about proxy objects last yearish but haven't intentionally used the pattern in my own software yet, so it doesn't come to mind immediately as a solution. Gotta find an excuse to apply the concept in my own code sometime so the idea sticks
Yeah proxy objects are one of those things that make a lot of python magic happen. They can be useful, but they can also be confusing for users sometimes.
10:05 I think there's a potential problem here with using a mix of explicit and packed arguments.
The explicit arguments will be absent from the packed arguments, so then would also be absent from the super call. Eg if the next sibling class also needs the argument `validators`, it won't get it because here `kwargs` doesn't contain `validators`.
If, however, you preserve all the arguments in the super call, then there may instead be a problem if the base class doesn't accept the extra arguments.
I am using this frequently but this is far the best explanation .
#pycharm
Nice video, though you didn't fully specify the MRO search algorithm (C3 linearization). I fully agree with Mark Lutz (Learning Python) on the concepts of "Cooperative Inheritance": best not to do it, since any programmer using a library you write using cooperative inheritance can break your code by simply adding a new mis-behaving class in the wrong level in your hierarchy.
wow. it's amazing that this language ever computes anything
I did use it once, it frelled up my code... but what it did for me was to prove that I should not have used classes for that problem in the first place. The need for "super" is a super-sized warning flag that the design is absolutely flawed.
Thannks, You make difficult topics interesting and easy to learn
Always wanted too see info about super but never thought about it directly. Thanks! #pycharm
This was incredible. Thank you!
Well, we never pass self as an explicit argument anyway so it makes sense that using super does not have us do that. Retains the pythonness of it all.
I was just reading reading an article from Real Python about super() to clear some doubts and you posted a video about the same thing! #pycharm
Hi James, I just started to seeing your videos, and they are of great quality. I'm a Mathematician and Data Scientist from Mexico. I just wanted to ask how did you achieve that deep knowledge of Python? Is there a set of subjects or books that you recommend? I have studied some basic OOP in Java. Thanks
Hi, great video, you always do such in-depth research! Could you perhaps do a video on creating python bindings for C/C++? It can be rather useful but I have barely seen anyone discuss it seriously and thoroughly on youtube.
This channel has really helped me level up my Python skills. Videos like this are really helpful for people like me who want an “under the hood” understanding of the language. #pycharm
As a next step, try reading a bit of the CPython source code!
This channel is basically Scott Meyers for Python.
That's some wonderful cursed knowledge right there, thank you.
This video made me realize how much further I have to go. I barely understand 20% of this
The important lesson here is that when MRO starts knocking on your door, it's best to just refactor your code to not use convoluted inheritance trees. Multiple inheritance is a mistake, and while I don't berate Python for supporting the use-case, I have yet to encounter a problem where the code couldn't be refactored to be more clear, without multiple inheritance.
Thanks for this video, I will use this in my current project #pycharm
thanks for clearing this out! Was confused about super until now #pycharm
16:10 This might sound nitpicky, but I think the wording you used here ("contains the class that is currently executing") is a bit incomplete and could mislead people who don't know better into thinking that using the ___class___ _attribute_ does the same thing. I would rather define what this ___class___ variable contains as "the class that the current function is defined under", or something similar.
In my opinion, if you need to think about MRO, then it's time to refactor your code. I use inheritance only when I create interface. Composition is the answer most of the time...
I noticed that you have a bit of background noise on your microphone. Have you tried out using noise suppression, such as that provided by OBS's built-in Noise Suppression filter, or Krisp's free desktop software?
Yes I use obs's built-in noise filter. I haven't changed my setup in a long time but many people are saying there is noise now. I double checked everything is turned on and setup as usual but I haven't diagnosed the issue yet.
Finally i understooood! Great explanation!!!