Python's most DISLIKED __dunder__ (and what to use instead)
Вставка
- Опубліковано 5 чер 2024
- What is your least favorite?
What is the most universally despised _dunder_ method in Python? It's __del__! From it's non-obvious relationship with del, to it's exception-ignoring behavior, to the uncertainty of whether the interpreter will call it 0 or multiple times, it's a real mess!
― mCoding with James Murphy (mcoding.io)
Source code: github.com/mCodingLLC/VideosS...
with statement docs: docs.python.org/3/reference/c...
dunder del docs: docs.python.org/3/reference/d...
weakref docs: docs.python.org/3/library/wea...
Plus equals video: • Python's sharpest corn...
Dunder new video: • __new__ vs __init__ in...
Metaclasses video: • Metaclasses in Python
SUPPORT ME ⭐
---------------------------------------------------
Patreon: / mcoding
Paypal: www.paypal.com/donate/?hosted...
Other donations: mcoding.io/donate
Top patrons and donors: Jameson, Laura M, Vahnekie, Dragos C, Matt R, Casey G, Johan A, John Martin, Jason F, Mutual Information, Neel R
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding - Наука та технологія
I would love a video about weak references! Especially about how they are implemented, and what ___weakref___ is for.
As I understand it, "weakref" is an overly elaborate term for simply a reference, that is, a pointer. But it has to be distinguished from conventional python references, which incorporate the reference counting mechanism that supports automatic garbage collection.
@@Graham_Wideman I haven't delved much into the details of python, but in Lua, a weak reference is a reference that is not counted for garbage collection. So, if something only has weak references, it is garbage collected.
@@alan5506 So you're saying Lua's use of the term "weak reference" is the same as what I wrote for Python?
@@Graham_Wideman Yes.
@@Graham_Wideman A weak reference is more than a pointer, because it needs to know, if the object still exists. Technically, you have the object itself and the reference counter (which is also an object managing the number of strong and weak references). The object is deleted when all strong references are gone. The reference counter is deleted when all (strong and weak) references are gone. Hence, a weak reference also "incorporates the reference counting mechanism" and is in fact a "smart" pointer, just like strong references.
My favourite is __mifflin__()
Gold
Does it return "This is Pam"?
This sort of video is one of the reasons I like this channel. I greatly appreciate explanations that clarify confusing/subtle issues. It's a pedagogical approach I tried my best to implement when I used to teach, so I'm somewhat biased.
WeakRef Video When
weakref video when
When weakref video
WeakWhen Ref Video
WhenRef Video Weak
Videoweak when ref ?
coming from C++ I naturally tried to write clean up code in __del__ but sometimes the modules were already deleted which ended up being endlessly confusing
Good video probably gonna save a lot of people from that specific headache
In the exact same boat! As I've always said, deterministic destruction is a drug, once you have it you'll look for it everywhere.
I learned programming with C++, but my career took me to higher level lenguages. Destructors are, easily, the feature of C++ I miss the most.
I didn't know __del__ was this hated. I have used it *once* before to create a scoped timer and it worked for that use case. Thanks for the insight on why I shouldn't lol.
6:10 I found another subtle case of this, where my ‗‗del‗‗ method had some local functions to do various related cleanup. Turned out that a direct reference from an inner function to the “self” argument of ‗‗del‗‗ would magically cause the object to be resurrected. I changed it to explicitly pass self as an argument to the inner routine, and that allowed the object to go away properly.
Moral: reference cycles are subtle things ...
Closures are distinct objects and they hold counted references to their upvalues, _if and only if_ they continue to exist after __del__ is executed it makes sense that they resurrect the object. If they don't, that's probably an interpreter bug or a completely braindead spec because it implies that either the count is checked before the locals of __del__ are collected or resurrection depends on whether the count was modified, not whether it's greater than zero.
They were executing _during_ del.
@@lawrencedoliveiro9104 This could very well be an interpreter bug then. Is it open source? Can you share a sample? I'd like to study this.
@@lawrencedoliveiro9104 To be clear, if the only reference to the closure was within the locals of the calling function that weren't upvalues, it should not count as a reference cycle, because the locals of a function that aren't upvalues die when the function stops executing. The names of upvalues are known statically, so they aren't supposed to count as references to the entire set of locals but the specific one.
@@tacticalassaultanteater9678 This was in my DBussy library.
2:26 Just a note that reference counting is a specific feature of the CPython implementation. There are some other Python implementations (such as those built on the Java VM) that do not do reference counting.
Probably the NullPointerException will hit you just as often as in Java
I think that in CPython it does both reference counting and the generational gc, but I could be wrong
@@codeofdestiny6820 yes, gc is for avoid reference cycle in cpython
@ I don't understand your answer, can you elaborate?
@@codeofdestiny6820 in the reference count there may be cases where A references B and B references A generating a reference cycle A B, depending on the implementation: 1 - the count of A never reaches zero, or 2 - it reaches zero but A and B leave the count however without deallocating memory and become inaccessible. some languages use strong and weak reference to solve this, A strong references B and B weak references A, A B (bad illustration in asc), when the count for A goes to zero, B will be automatically deallocated by have a weak reference to A (swift and also is posible in cpython but not is the default). cpython uses the garbage collector to solve this, because if A and B become inaccessible without deallocating memory, the garbage collector will solve why it knows the inaccessible objects (I don't know the details it but this is the idea)
Thanks for clarifying this. I now know that I have some code to clean up. I had my doubts that I was doing it right. It currently works as it is used but I can now see where it will fail.
Funnily enough, I discounted context handling as unrequired but since adding `__enter__` and `__exit__` will cover the bases, I guess I'm going implement them.
You have the most informative python programming videos I've ever seen. Really love your content. I find tidbits like these very helpful. Thanks!
LOL - The comment at the end about hitting the like button an odd number of times made me laugh.
Great video. Thanks.
Glad you made it all the way!
Thanks, one more time, for putting time and effort into making these videos!
I've taught Python for beginners myself, and made sure to make my students aware of your content ever since I found your channel myself.
Since you asked: I'd be glad to see a follow up video on weak references sometime in the future.
I absolutely want to see a video about weakref: when it's the best situation to use it, when it will not solve my problem but I would assume that it would, and how to break it. Thanks!
Wow, this channel is pure gold. Thank you so much for your videos.
Glad you like them!
8:08 On Linux, it is possible to create temporary files that have no directory entry in the filesystem. That means they automatically disappear when closed.
How to do this? Please share
O_TMPFILE option to open(2).
@@illegalsmirf You can delete open files in Linux. You can read/write them, and pass open file descriptors to them to children.
This brings back memories of the RAII idiom (Resource Acquisition Is Initialisation) in C++. I remember (wrongly) assuming that Java finalisers worked in the same way as C++ destructors. Java's implementation of finalisers (finalize method) is very similar to what you describe here with __del__. The semantics of destructors in C++ was very simple and predictable -- much akin to context managers in Python.
We have some legacy C++ code with initialize/finalize methods. It’s disgusting, whoever wrote it was trying to write Java code in C++.
So __del__ is like finalize() in Java. (Sorry about the italics eating one of the underscore pairs; UA-cam doesn't seem to provide a way to escape it...)
Unpredictable, and strongly discouraged as a means of resource control or critical cleanup.
Though may be usable as a fallback if the programmer forgets to close properly and that will result in a critical resource permanently lost within that execution. Or worse, silently incorrect behavior.
In which case, you should first try to redesign so that isn't the case.
If it is unavoidable, then at least raise a warning about the misuse or a stacktrace.
(And also like finalizer, reference type objects are the superior solution for this anyways)
Slapping (self) on the end of the dunder name helps.
__del__(self)
A programmer should never do something silent as a fallback for if they forget something. If they forget something the program should tell at the programmer loudly and throw an error.
@@minerscale In pure best practices ideals, yes, I agree. Fail on it.
However, there are plenty of people/organizations who won't follow/followed best practices. And won't (or won't be given time by their higher ups to) learn those best practices and refactor accordingly.
If it is an existing code base you are adding this check to, then businesses will not like you "breaking" their application/stack, even if what is actually broken is their code.
Or even if it is a new API, there is a good chance developers will find this strictness too annoying and/or time consuming. and instead chooses (or the higher-ups choose) some competing product. Instead of doing the right thing and fixing their own coding practices.
It sucks, but here in the real business world, not accounting for or providing avenues to deal with less than ideal development circumstances (or outright laziness) will cause friction
I think the best solution is configurable behavior, "fail", "warn", and maybe "ignore".
Kind of like how "strict mode" works on Android (if set, raises exception if a resource cleanup is missed, else it just logs a warning)
"this" had always confused me, so I'm glad you said that "self" can be used like "this". However, I had started to use "for this_item in items:" So now I can accept either one, and "self" will work fine in Python.
Awesome content, thanks James!
Thanks a lot for the video, the topic and explanations are awesome (as always)
It will be really nice to have a video about garbage collector (primary algorythm, is it "tunable", native extensions and GC)
would definitely like a weakref video :)
Great video! I found this channel recently and have been binge watching videos, and have found myself learning so much. Can you do a video briefly going over all the __dunder__ methods that can be implemented in Python? I find myself using the same ones in almost every project I work on and wonder if there are any ways I can improve my code by using different / additional __dunder__ methods. Thanks!
Working on one already!
Every time I think I know everything about python, mCoding finds another hidden feature I haven't heard about. Please do a video on the nonlocal keyword!
sweet upload mCoding. I broke the thumbs up on your video. Always keep up the very good work.
Very good to know. I have been using __del__ for a long time as distructors
Awesome video, thanks!
Excellent video as usual
I wasn't sure of what dunder method was the most annoying before this video. I am now. I'm also relieved that you discussed options to it in the end, my inner peace would be destroyed if you didn't so thanks for that.
First "click-bait" title I've seen for programming related content I've seen :D
Wow I've just never thought about __del__. On my teams we tend to agree to not overload any dunders except for new and repr. Thanks for making this video letting us know of the hole :o
Edit: I still think __del__ COULD be useful in very niche instances, but overall it needs to be insanely justified.
IIRC, the CPython GC detects reference cycles, but then drops all involved objects without calling their dunder del methods - after all, I might expect to be able to use self.parent in the child's dunder del, and vice versa, so in which order should they be cleaned up?
When a Python video triggers your C++ PTSD ...
Can you tell about dunders __reduce__ and __reduce_ex__? I guess they are old and used for serialization, django's sources customize them for some structures.
Not related to this video, but do you have any recommendations for PyCon US 2022 videos?
Watch the new typing stuff and the one on the history and direction of the match statement!
Great video! Can you make a video about yield from
I'd love to learn more about weak references
i believe the builtin lib atexit is also a good way to run cleanup code, if it's code intended to run once right before the interpreter exits. you `atexit.register(foobar)` to schedule `foobar` to be run on the process' shutdown. some other builtin libs use atexit like this
with the weakref module, doesn't this also mean there should be some kind of guideline about not implementing cyclic references directly? such as it being formally recommended that if children need to refer to their parents, that they use a weakref for it? i suppose this also has problems because if a child needs to refer to the parent then any additional references to the child will necessarily break if the parent vanishes, so they would need to have some sort of exception behavior where they act like "None" when they discover their parents are None
Do you have a video explaining what's CPython?
7:05 Could this potential issue be avoided with a "import shutil" line to ensure it is loaded, or is that still problematic somehow?
Excellent idea, but unfortunately trying to import a module during interpreter shutdown is likely to cause a crash or other badness for ... "reasons". You also shouldn't attempt to start new threads or subprocesses.
My solution to this is to register an atexit function in the containing module that goes through all classes that have potentially troublesome ‗‗del‗‗ methods and delete them from the class. E.g.
def _atexit() :
for cłass in Connection, Server, PreallocatedSend, Message, PendingCall, Error, AddressEntries :
delattr(cłass, "__del__")
♯end for
♯end _atexit
atexit.register(_atexit)
Instances that disappear during normal program operation get cleaned up as usual; ones that are left at program termination are not.
Ha! I saw this one coming. The best part about the unpredictability of __del__(self) is when you need to run async code cleanup.
Just don't. Find another way to work.
So basically __del__ can prevent memory corruption bugs that pops up every now and then in JS. (It is being called when the object is removed from memory)
After using rust for so long, and particularly Arc and Rc (ref counting stuff), del reads exactly like the Arc::drop non-inlined Arc::drop_slow.
do you prefer self.__class__ over type(self), and if so: why?
when I was trying to make a custom class divide and the dunder just wouldn't work... I was so angry when I finally looked it up and found out it was __truediv__, instead of __div__. like just why
Because div is ambiguous, Python has two divs, a/b and a//b!
Everytime I think I am getting good at python I see your videos and realise how much I don't know.
At first I used the __del__ method to free resources but stop using it because it was called twice frequently, so I replaced it with atexit. What is the difference between weakref and atexit?
It turns out weakref uses atexit under the hood ^^ so not much probably
Relying on destructors is generally a bad idea in garbage collected languages. Same goes with Java, where you can never be sure any destructor runs and objects can be resurrected. This is odd behavior, if you are coming from C++, which makes strong guarantees about object lifetime and destruction (see RAII). For cleanup code, like deleting temp files, I usually use atexit. Finalizers seem to be an interesting alternative, when they handle being called directly before exit.
Agreed and I know exactly what you mean coming from C++. Btw weakref finalize uses atexit under the hood.
Does anyone know what the other versions of 'cpython' would be? I've never heard of that term before.
He means Cython I’d guess
I think he meant other implementations of python per se.
CPython is an individual reference implementation, but there are a lot of other implementations, like IronPython, PyPy, Jython and less known ones.
@@deimuader ok thanks bro! Yea i guess I had always heard terms like PyPy and Jython before, but never actually connected it to what they really were. The fact that most python implementations are Cython makes a lot of sense as to why python can have some nice bindings with C. Thanks for the responses yall!
What cleans the finalizer?
Okay, just wow! I didn't know any of that. I always thought that "del x" called __del__
My prediction would’ve been get_attribute (not to be confused with get_attr which is amazing)
CPython isn't guaranteed to run the destructor when the refcount hits 0. If the object is marked "immortal" (None, True, False, and the like), then even hitting 0 doesn't do anything.
If the refcount hits 0, then it invokes tp_dealloc, as provided by the C-struct holding the python data. It is possible for _that_ to do nothing, but only if you are using types derived from some extension library.
The example shown at 4:00 should in my opinion throw an exception. Circular references are poison and should never be allowed.
And people used to complain about C++ destructors....
__ del __ is pretty useful for writing ctypes ffi/library wrappers
Yes. But also see my comment about atexit.
I appreciate being alerted to the issues, but this presentation would be more complete if it told what a finalizer actually does. As it stands, it appears that sprinkling finalizer dust over your code in random places casts some sort of protective spell on it that makes it all good.
At the end he does ask to comment if you want to know more about weakrefs, so I guess that was his cliffhanger and you would like to see that follow-up...
😉
A finalizer is for doing anything you need to do right before an object is garbage collected.
@@mCoding "A finalizer is for..." Yes, that much is obvious from the name. But in this video we don't seem to see where that "doing anything" is actually defined and how it gets invoked. I know, I could RTFM... just that it seemed like rather a hole in this otherwise nice story.
I want to see a video about weak references
So is the conclusion of this video that there are no circumstances under which one should ever use dunder-del? I know the creator here did specify a number of situations that could go wrong, but are there *any* circumstances under which one might want to use one?
"Dude, you might not get a __del__"
my least favorite dunder that i actually use? Well I dunno, but I kinda dislike the separation of repr and str. I usually just went straight to __str__, then just slapping def __repr__(): return str(this).
finalizers are idempotent!
I made sure to avoid using a math word in a programming video but you're absolutely right. Idempotent is the technically best way to describe them (or how they _should_ be).
As a java programmer, what on earth is a Dunder method? Like, from The Office?
Dunder == "Double underscore" refers to names in Python that begin and end with two underscores, like in the video title.
This whole thing is a problem all gc languages at some point face. Java's finalize() method is equivalent to __del__. Java also has try(..) blocks which are equivalent to with.. as.. : java also have the cleaner class which is equivalent to finalizers in python. This shows that even though java and python have little in common besides being gc languages they still share this exact problem. Java actually is in process of removing the finalize() method (__del__) in the near future.
The dunder blunder
Whoever wrote __del__ must...(ahem) a dunder head!
...I'll see myself out
3:20
As far as I know, this is outdated documentation. Now it is always called at the end of the program when all objects are destroyed.
I'm enjoying reading through all the comments of people trying various ways to type ___dunder___ methods without UA-cam messing with the formatting 😂 (Fitting that it took me 3 attempts to get this comment to format like this)
x.children = [y]
y.parent = x
that's what weakrefs needed for
Indeed weakrefs can break this cycle, although CPython 3.4+ is able to detect and delete these cycles so it is not necessary. Though i will certainly talk about this if i do a weakref video!
Weak refs are useful.
But why does Java need _three_ different kinds of weak refs, when Python can make do with one?
@@lawrencedoliveiro9104 I don't know exactly, but Java in comparison with Python has GC (Garbage Collector) that defines behaviour of working with each type of reference (Soft/Weak/Phantom). This behaviour can also rely on jvm realisation (e.g., _as I know or think_, Hotspot uses timestamps for SoftReference and IBM VM counts how many gc calls this ref survived).
I didn't find any useful material to put here, so I wish you good luck with understanding refs in Java
At 4:20 (heh), you say in the comment that the refcount is 0!, but it's not. It's actually 0.
so it's like a desctructor but doesn't work well
3:09 > _"circumstantial"_
nice term for such cases.
not impliable...
So, what is del for then?
Dunders!!
I have never used that keyword and I am not sure why I would.
_ _call_ _ and _ _new_ _ would be in my top 3, but _ _del_ _ is the king of Dutch dunders (not be obvious at first unless you're Dutch).
Every time I even consider leaving my much beloved C and using Python I run across another reason to just go back into my glorious C shell. If a program wants/needs to delete a THING, then delete the thing, and quit expecting the compiler or OS to hold your bloody hand. But, then again, still don't see the point in moving from C to C++, nor does that guy who wrote the Linux Kernel.
The whole idea is that it eliminates a whole class of errors. Think about it, if you have an API where you create a thing, and you need to destroy that thing (else get a resource leak, or maybe other bad things happen), and you expect the users of the API to clean it up... wouldn't it just be easier if the language had a built in feature to make sure you can never get that wrong? It lowers the complexity of APIs, because there is no additional function to clean things up, it's tied to the object and (more or less) never gets called manually) Now you just declare a thing, and when you're done with it the cleanup code gets run, either via garbage collection + a callback, or in C++ you have RAII, which in turn is the basis for OBRM in Rust - the latter two being completely deterministic.
It is the single feature from C++ that I love so much that I could never use C, because it doesn't have that.
i like how u have a reference cycle in "parents" and "children"... *sweet home alabama*
jkjk
I love Dunder Mifflin. Does that count?
It would only count if you disliked it!
Comparing to other languages that include cleanup code(defer in go, deconstructors in c++), the caveats for dunder del seem all very reasonable. You SHOULDN'T be doing long running operations, opening files or reviving variables in cleanup methods. That's just inviting trouble! I know it's usually a bad argument to say "the programmer should do the Right Thing" for language features, but on an implicit cleanup method like dunder del it's one of the places that there's only so much the compiler/interpreter can do. At best, python could warn against certain patterns but they could always be circumvented.
Tbh if you are relying on such implicit behavior for cleaning up, I'd say you need to rethink your code(like this video proposes), but I don't see any reason to hate dunder del because all arguments "against" it are pretty obvious for anyone who has ever dealt with memory management(and if you haven't, you're probably not the target for dunder del in the first place...)
As an addendum: Go's _defer_ can actually be used for all the things above, since Go has a different type of garbage collector and, as a compiled language, it actually just rewrites your function to execute _defer_ at the end of it instead of at the point it was called, no different than try/finally statements. Of course, Go has a runtime.SetFinalizer which is more closely related to Python's dunder del, but that one is far more explicit and doesn't attach itself to every instance of a struct(though technically you could write a constructor to do just that). The exact same caveats as dunder del applies to it.
Yay for mentioning context managers! Nay for not calling them "context managers"...
I mean... pretty sure someone who searches for "with statement" will end up at the right place, and it is arguably clearer what a "with statement" is referring to than a "context manager" to someone who doesn't already know what with statements and context managers are and how they are related.
What does the assert keyword do?
to perform an assertion on a condition, an AssertionError would be raised if the condition evaluates to False.
It is really useful when you're writing tests
You can add a message to that Assertion Error by providing a second argument, e.g
assert a!=b, "Oh dear - a shouldn't equal b"
Throws an error if a condition is false.
Be careful with asserts, as they can be optimized away if you start a script with python -O script.py.
Lol i didn’t hate __del__.. but now I do
I know that this will be basically considered blasphemy, but I wish you did videos on PHP. I don't even use python! I just enjoy your videos!
Maybe one day... not today though!
Weakref video please.
can't we avoid all of this with dataclasses package?
This mechanism is still needed for things like wrappers for non-Python libraries. They tend to expect explicit calls to dispose of objects, and it’s nice if the Python user doesn’t have to worry about such things. It takes a certain amount of mechanism to ensure this happens reliably.
Weak ref vid, plz!!!
repr is awesome
If you know you know!
slapped 3 times
who was the dunderhead who added this?
i use del :)
I have to say my favorite one is " if __cats__ > 1 then __ __ __ __cats__ == 'ho!'"
'__setattribute__' because it desen't exist, which makes me crazy as you get: '__get__', '__set__', '__getattr__', '__setattr__', '__getattribute__' and.... None
Laughs in COM.
Watching your videos just proves to myself that I don't know anything. 🤦
Ok you convinced me it's bad, but why don't they deprecate it?
Dunder Mifflin
the worst is the word 'dunder' itself
If __del__ is so unreliable, why does it even exist? Who uses it?
It’s not so unreliable. It can be useful -- with care.
C Python? Wait there is more than 1 Python?