I found that using Pydantic data validation together with a more functional programming style leads naturally and organically to a "let it crash" error-handling pattern. Most of the applications I implement are data-driven and data-intensive, and the majority of the errors happen because data is not well formatted or some unforeseen data behavior. Using Pydantic, I can catch those errors as upstream as possible and avoid spurious computations that might lead to unexpected run-time errors.
I've been leading a migration at work of our microservices from Python to Golang. At first I was afraid of the whole errors as values thing but I'm now a believer. It just straight up leads to better error handling. You always have precise control of the flow, no implicit exception ignoring, no nested try except mess. I'm kinda baffled now that try except is the most common error handling paradigm to be honest
This is my thoughts exactly. Since I dived into Rust I find it actually more difficult to write Python, because I have this constant uncertainty. Will this function in my long running program fail? Can it even fail? If so, what type of exception will it raise? In Rust I know I only have to worry about fallible functions, and I have clear information how and when they fail because I can follow the code. And I can decide on spot if I want to handle the error or propagate. And as a side-effect of Rust's functional style I have the guarantee that no state has been altered when error is propagated. If I wanted to achieve that same thing with with try-except I would have to at least write many more lines of code and I still wouldn't have that much of control. So counterintuitively I just find it easier and faster to code in Rust than Python because I know exactly what happens in my program. And also... why the hell is ctrl+c an exception?!!
I have a Python application in production, and it has been without any errors for over two years. The secret is "tests" and logging 100% of unhandled errors. Every unhandled error is on the dashboard immediately. It was fast after going into production. All unhandled errors were handled. The language and type of error handling that language offers is not important, when your error handling is simply bad. But for REST API golang exception handling is very comfortable.
@@cozajeden I'm not arguing you can't have exhaustive error handling in Python, but it's not the most comfortable for me. With static typing I don't need to handle type errors at all. Then, if for example error handling happens a few levels higher in call stack, I can just add "?" at the end of a fallible function calls and propagate the error to more convenient place without loosing any information. With well defined error types I can even have a full information about the error origin within call stack. And after that the main thing that is left for tests is to check the code logic. I am honestly very impressed by people who can work with big Python codebases, because I myself can't. But personally after learning Rust I find writing Python uncomfortable, because I have to worry about multiple things that Rust handles for me.
This highlights why errors-as-values is generally a better error handling paradigm. It can be very difficult to tell when python code (especially from a library) might raise an error.
How are error-as-values supposed to even work? If a low level function returns an error, but only top level function/controller is supposed to deal with the error, you have to pass it up trough every function. Exceptions do that automatically until you catch them. If a library throws a new error, you let the code log it, and you later fix it by handling at the top.
@@rain-er6537 Exception do exactly that but this is exactly the cursed and bless we are talking about. With Exception you are never sure if someone down in the code will make an error like forgetting to check Json and create a reaction of error. With error has value you explicitly agreed to always acknowledge your possible defunct function can return an error and what to do with it. Of course you can think if err == nil in golang are atrocious but don’t forget that’s the point. We don’t want to forget our error, we either prefer to panic or handle it. And more, don’t write unit test for the sake of maybe my code can throw, that’s overkill.
@@rain-er6537exceptions do that but explicit is better than implicit. Passing it up as values is explicit and when you have it monadic then it is just a small number of characters to take care of that propagation
Rust's Result type is the best paradigm. It forces you to do something with possible errors, either handling them or returning then to your caller. Forgetting to handle them is a compiler error.
@@JayKnight I think the important part of what makes Rusts errors so good, far better than other error as value approaches, is that it still allows for the common case of "just pass the error onward" to be handled in an ergonomic way. There's nothing more ugly to me than Go code full of "error handling" that is emulating exceptions by bubbling up errors (return nil, error)
I must say that „4. Focus on real issues“ at 7:28 is the most important point. Especially when you say:“You first deal with those which happen most offen.“ In my opiniin that was the best shot into the bullse eye (term from dart :-)) I enjoyed the rest of your ideas also. This was the best UA-cam post since a long time full of correct and wise best practices instead of garbage what UA-cam is full with. I will consume more of your vids upon now for sure. 👍 Best regards your Nejat „Gaaaanz liebe Grüße aus Deutschland“
You hit the nail. In my initial Python coding (15 years ago) I was trying to deal with any possible error within any function - even when there was nothing what could be done within given context. My "careful" coding became never ending effort. With the Let it burn approach the relieve was enormous - similar to when you throw away a backpack full of responsibilities you shall not bear. Suddenly the coding became a fun run. Thanks for explaining this.
Fail fast! Python is particularly well suited for this approach that I employ as well. The real enabler is the "last chance" logger feature. In reality, robust error handling is an art form where skill only really improves with experience. Great topic! You covered it well.
YES YES YES!!!!!!! For the life of me I can't understand why people write so many useless try-except blocks in every function that just don't help anything and make the code worse... and often also swallow exceptions they really shouldn't... I can guess this video idea came from doing consulting work and seeing such code...
The only time try-except blocks make much sense to me is when the error is an error that could happen even if I know my code is bug free (for example, a resource missing error).
It really depends on what those try catch are used for. Try catch for logging and printing is just retarded but to run logic on failure to handle stuff gracefully is the way to go.
Fail fast is important with few caveats in mind as you said. I would insist in "Exception are for exceptional behaviour", sometimes handling functionally the issue when it is business related is a better idea. Code has clear typing modeling what can or cannot exist, and conditional are way faster than exception.
it's a good approach for some cases imo i think no one can escape errors or predict them and also for more complicated code you would end up with a bunch of ugly "if" statements and we back to zero, but I like the idea of making the obvious errors to not be an error and make the process move on it's brilliant, we don't have to raise errors if we can handle it properly.
Nice advice! I follow the same practice when possible. Even still, when errors become visible, you should not start shooting try/catch everywhere, you catch the error, at a point where you can do a proper handling that makes sense at that point of the code, according to the status of the project, to what user's expect, to what you want users to do in that case. Don't make you're code unreadable with insane try/catches specially at points where you don't even even know what to do with the errors.
Literally had this case today at work, though kinda the opposite...! I need to apply a database update for a bunch of items, but one of them was failing. I had left all the error handling to the outermost layer, such that the error was causing the whole database tramsaction to fail. Easy to therefore find and diagnose, but i realised i really wanted the error handling to allow other items to pass still. By having the "let it fail" principle, i could therefore choose the point at which i was prepared to handle the exception because i didn't want to fail at that point, but to log and then proceed with the next item. It makes for much more sensible error handling at the point where the error could be properly processed, not constantly logging it until someone knows what to do with it. Other parts of legacy code does do that 'print and raise' handling, and it becomes hard to actually work out what errors I'm actually trying to handle and to get the severity/regularity of them
In the Web API context, a favorable approach is to avoid exceptions in the sense that is best to return a "union" Result type that may contain either the correct value or the error that happened. Exceptions might slow down considerably the calls depending on volume of calls.
@@dynamicgrad3820Errors as Values are without a doubt, better. It's alright to use exceptions in python for building little automation scripts; but for web APIs and stuff, languages like Go and Rust win with explicit errors as values. You can't ignore the error implicitly there, you choose either to crash or to handle (or maybe ignore sometimes). There are no exceptions in Go, if you try to connect to a database, there are 2 values returned and you need to check if the error value is there or not. Rust did it better imo.
@@dynamicgrad3820 there is always a high-level exception handler as a middleware for unpredictable cases. This cannot be omitted obviously. But most exceptions such as when an entitity is not found, are not "real" exceptions.
"Dead programs tell no lies". ("The Pragmatic Programmer: your journey to mastery" by Andrew Hunt and David Thomas). I agree with this philosophy. Often there is a mantra that the program "must not crash". Except at the very top level, there are often no meaningful recovery actions, dump the transaction (or run of the program) and start afresh. Otherwise the implementor, via their code, has 'lost control of the application's internal state' and all results are then suspect. Thanks for your insights and this video.
Great to hear a bit of sense on the matter. Similar to seeing junior engineers using dict.get, i ask why and they say 'incase the key isnt there'. Great so now we just get a type error 2 lines down when you try and add an int to None
im confused, if using dict.get, wouldn't you do a check right after to make sure the result isn't None? or i guess it might be better to let it error out in order to fix the code to ensure the dict always contains that key.
I fail fast and loud while also tolerating unexpected things from system integration, log and raise. Another good way to design your own architecture is to return empty native types. e.g. if we were expecting a populated dictionary back we can try: / except : return {} callers or components can then still check using "if not"
I deal mainly with program automation, where the users don't know much about how the scripts work, and can't provide good feeback, and will simply stop using them instead of letting me know. The scripts are also relatively small, so logging is not practical. I've made it standard to include quick tests i.e. if array.len== 0 return None or if mxs.superclassof(selection) !=geometryclass return None. Any returned None than triggers a pop-up t alter the user to the source of the problem. I apply the same structure in program specific scripts (maxscript) for a consistent behavior and users can actually let me know why it failed, so it can get fixed.
This may be fine when you yourself are the user of your software, like your backend example, but a lot of software doesn't have the same luxuries. An end-user app that adopts "fail fast" usually leads to "uninstall fast" and lost business forever.
Say that to Nasa which has sattelites running for over 30 years and every code base in Nasa is made with fail-fast approach. They even released a paper about it
In case of key errors I prefer to create data structures like dataclasses and manage errors at instantiation level usign invariants or a similar mechanism.
Some people at my company tend to think they need to silently catch exceptions and return null/None. This makes us end up with code that swallows exceptions and it takes a lot more debugging to find out where the issue is than if they had just let it crash and burn!
What would you do when you need to search for several files and want to inform the user that the 2nd, 4th, and 5th files were not found? If you raise an exception, the program will stop running when it doesn't find the second file. I'm a beginner programmer and I'm looking for a solution
@@dynamicgrad3820One thing you can do in this case is use the try catch blocks to catch the error and if this error occurs you can append the file along with any useful information about the file in a list, kind of like a dead letter queue. Try to be specific with the particular exception you are catching, you want other error types to crash and burn. Try catches do have their uses, we just shouldn't use them without any good justification for it. Thought, if its just searching for a file in some directory, there's probably a way of searching and returning a bool, if the file exists or not, rather than an error.
@@dynamicgrad3820 Letting it crash and burn is for when the program is in a state where it never was supposed to be and was never intended by the programmer. It means the code is broken. This is different from a user error. If it expected in your case that certain files could not exist, you can return some kind of result object instead of raising an exception. And let the user know what they can do to solve the issue.
@@dynamicgrad3820 sometimes it's better to just have an if statement and do like: if os.path.exists(path): do something else: log into a list , then output to the user that XXX files were not found if list is not empty. Yeah, you could do the same thing with a try/except block as well, but something like that i'd end up just using a if/else statement. edit: unless you don't know the directory path of the files up front to begin with and need to do something like a os.walk() to iterate through directories/subdirectories to find a file, then if it returns None, log it into a list and output to the user it couldn't find it. I've had to do this kind of searching for files with numerous programs.
@@dynamicgrad3820 This is very trivial - you can use chatGPT or any LLM. It helps a lot if you have a simple issue. files_to_find = set(...) files = set(os.listdir("some_directory")) missing = files_to_find - files if mssing: raise FileNotFoundError(f"missing files: {missing}") You can do it with a for loop or ith list comprehension too. Although I'd focus more on tutorials, leetcode, just to get the basics.
I don't really agree with this video (apart from the caveat section which seems is at odds. As the owner of a company developing an iPaaS platform providing pretty heavy weight integration with ERP, Payment, eCommerce, and whatever systems, you really want to augment the errors so users are informed what went wrong and where - in sensible language they & understand. Not only does it give your users an idea of the error, the cause and maybe a hint as to the resolution, but it also gives your support and development team. You probably want to retry transient errors (such as retrying webservice calls). Do you want to fail fast when you can wait 500ms and retry the operation where it will likely succeed? And if need to fail, you want to fail gracefully with as much info to your logs and users as possible.
12:48 Using "except Exception" will not prevent syntax errors, as the interpreter identifies those before the code even runs. It would catch NameErrors, but those are trivially recognised statically by any linter, so it's not a big deal. It's still a decent way to write a catch-all at the top of the program. On the other hand, one should _never_ use a bare `except:` as that would catch many wrong things (e.g. it would catch Ctrl-C and prevent you from stopping your program, which you almost never want).
You forgot a 3rd option. You can create a decorator function for handling all exceptions and then just add the decorator to whichever function(s) you want. That way you don' t need to encapsulate code with try/except blocks everywhere and keeps the code clean. You can have the decorator returns things like the traceback, the function name where the failure happened and the exception. For Example: def handle_exceptions(function): """ wrapper/decorator for handling exceptions and returning the traceback of what caused the error and return value(s) of the function """ @wraps(function) def wrapper(*args, **kwargs) -> tuple: try: result_values = function(*args, **kwargs) return (None,) + result_values except BaseException as e: # extract traceback details tb_str = traceback.format_exception(etype=type(e), value=e, tb=e.__traceback__) error_info = ''.join(tb_str) return error_info, return wrapper Then just add @handle_exceptions decorator to whichever function you want.
This is bad... Firstly, you don't want to catch BaseException. Secondly, it may cause your code run longer than it should, causing a different error. Like a whole video is about it. Thirdly, it's easy to forget what exactly this "handle_exceptions" does and use it on a function that returns nothing, or worse, in some case returns nothing. Handling exceptions on top level is way better.
@@Plajerity this is not bad and there is nothing wrong with catching base exception under certain circumstances. It was just an example you don't have to use base exception. I had to use it in my example for some code because an error I was receiving was not being captured by any exceptions except base exception. I don't see how it would be easy to forget what it is based on the name of it. It doesn't have to return none if no errors , it's just an example. Im not sure what you are going on about with it making code to take longer, because it doesn't. Also, you can't always handle exceptions at top level, especially if you are dealing with multiple processes or threads.
@@resresres1 Maybe under certain conditions you can catch BaseException, but I definitely woudn't make it as decorator. Just one-time use. Ideally you shouldn't use it. I didn't mention you modify the function output, making it more complicated. You're right about multi-threaded workload, sometimes it's an issue. I have too little experience to say anything constructive... In such cases I prefered to return an exception and catch it as soon as possible, checking if it's an exception instance.
Of cause you can also just chain the exceptions you want to handle and then raise a generic exception for every thing else. You can do that in a single try except block.
The only reason i ever use the try .... 'error {e} occured' is if i have some script that runs perfectly fine for 99% of the time and i need it to continue after some exotic exception . I have automated a browser game and i need the script to run for long periods of time for example . Only at the top level and probably a temporary solution.
Nobody: ...... UA-cam Coding Channel: "I debug by letting my code crash and burn as soon as possible" Crowdstrike Engineers: 👀😐 Fatigued future crowdstrike engineers who watched this video: "I debug by letting my code crash and burn as soon as possible" World war IV starts the following day 💀
A good film, but I’m wondering about a situation where a function should provide more than one piece of information about an error. For example, if I'm looking for three files, I would like to inform the user which of those three files could not be found. In that case, using try-except at the highest level is not able to catch all the errors.
12:17 Bare except clauses are problematic, but this is not one of them. A bare except clause states no exception class at all. This catches even SystemExit and KeyboardInterrupt, which is why it’s so problematic. This is, however, “catching to general of an exception,” which is a problem for the reasons discussed.
oh damn, i just typed this but didn't see you already had mentioned it. I have an exceptions decorator for this exact thing that gives me the traceback, function and the error.
Imho java-style exceptions are a failed experiment. Rust-style errors- are- return- values works well in practice, but only because there's syntax and macros to hide the plumbing. Pannicking is a last resort, as in a many thread or many process application it leaves you in a what the hell state.
God I hate those `except` blocks saying `print(an error happend .... {err}); raise`; what's the point? Just remove it and, as you mentioned, make the code much easier to follow
@@tuobgWhy would you need to parse your logs easier? When I parsed some big files I knew ahead what values I'm looking for, pretty easy to find lines with grep. You could use a custom error I guess. I don't get it how standarizing error mesages could help.
chatgpt code tends to do that type of error handling you showed near the beggign, it does nothing, makes code harder to maintain and gives you less information
When I was a child and learnt that the ocean was coming to the Netherlands, I was shocked and couldn't sleep for fear that the Dutch would be flooded and all die. I still worry about that to this day. P.S. I'm glad you survived, Arjan.
Assert statements are ignored in compiled code. They shouldn't be used for guard clauses in production code, they should only be used during development.
Learn how to design a piece of software from scratch with my free Software Design Guide: arjan.codes/designguide.
I found that using Pydantic data validation together with a more functional programming style leads naturally and organically to a "let it crash" error-handling pattern. Most of the applications I implement are data-driven and data-intensive, and the majority of the errors happen because data is not well formatted or some unforeseen data behavior. Using Pydantic, I can catch those errors as upstream as possible and avoid spurious computations that might lead to unexpected run-time errors.
Interesting.
I've been leading a migration at work of our microservices from Python to Golang. At first I was afraid of the whole errors as values thing but I'm now a believer. It just straight up leads to better error handling. You always have precise control of the flow, no implicit exception ignoring, no nested try except mess. I'm kinda baffled now that try except is the most common error handling paradigm to be honest
This is my thoughts exactly. Since I dived into Rust I find it actually more difficult to write Python, because I have this constant uncertainty. Will this function in my long running program fail? Can it even fail? If so, what type of exception will it raise?
In Rust I know I only have to worry about fallible functions, and I have clear information how and when they fail because I can follow the code. And I can decide on spot if I want to handle the error or propagate. And as a side-effect of Rust's functional style I have the guarantee that no state has been altered when error is propagated.
If I wanted to achieve that same thing with with try-except I would have to at least write many more lines of code and I still wouldn't have that much of control. So counterintuitively I just find it easier and faster to code in Rust than Python because I know exactly what happens in my program.
And also... why the hell is ctrl+c an exception?!!
I have a Python application in production, and it has been without any errors for over two years. The secret is "tests" and logging 100% of unhandled errors. Every unhandled error is on the dashboard immediately. It was fast after going into production. All unhandled errors were handled. The language and type of error handling that language offers is not important, when your error handling is simply bad. But for REST API golang exception handling is very comfortable.
@@cozajeden I'm not arguing you can't have exhaustive error handling in Python, but it's not the most comfortable for me. With static typing I don't need to handle type errors at all.
Then, if for example error handling happens a few levels higher in call stack, I can just add "?" at the end of a fallible function calls and propagate the error to more convenient place without loosing any information. With well defined error types I can even have a full information about the error origin within call stack.
And after that the main thing that is left for tests is to check the code logic.
I am honestly very impressed by people who can work with big Python codebases, because I myself can't. But personally after learning Rust I find writing Python uncomfortable, because I have to worry about multiple things that Rust handles for me.
This highlights why errors-as-values is generally a better error handling paradigm. It can be very difficult to tell when python code (especially from a library) might raise an error.
How are error-as-values supposed to even work? If a low level function returns an error, but only top level function/controller is supposed to deal with the error, you have to pass it up trough every function. Exceptions do that automatically until you catch them. If a library throws a new error, you let the code log it, and you later fix it by handling at the top.
@@rain-er6537 Exception do exactly that but this is exactly the cursed and bless we are talking about.
With Exception you are never sure if someone down in the code will make an error like forgetting to check Json and create a reaction of error.
With error has value you explicitly agreed to always acknowledge your possible defunct function can return an error and what to do with it.
Of course you can think if err == nil in golang are atrocious but don’t forget that’s the point.
We don’t want to forget our error, we either prefer to panic or handle it. And more, don’t write unit test for the sake of maybe my code can throw, that’s overkill.
@@rain-er6537exceptions do that but explicit is better than implicit. Passing it up as values is explicit and when you have it monadic then it is just a small number of characters to take care of that propagation
Rust's Result type is the best paradigm. It forces you to do something with possible errors, either handling them or returning then to your caller. Forgetting to handle them is a compiler error.
@@JayKnight I think the important part of what makes Rusts errors so good, far better than other error as value approaches, is that it still allows for the common case of "just pass the error onward" to be handled in an ergonomic way.
There's nothing more ugly to me than Go code full of "error handling" that is emulating exceptions by bubbling up errors (return nil, error)
I must say that „4. Focus on real issues“ at 7:28 is the most important point. Especially when you say:“You first deal with those which happen most offen.“
In my opiniin that was the best shot into the bullse eye (term from dart :-))
I enjoyed the rest of your ideas also.
This was the best UA-cam post since a long time full of correct and wise best practices instead of garbage what UA-cam is full with.
I will consume more of your vids upon now for sure. 👍
Best regards your Nejat
„Gaaaanz liebe Grüße aus Deutschland“
one more tip, if you log an error or print it, use repr(error), because that will also include the type of the error
You hit the nail. In my initial Python coding (15 years ago) I was trying to deal with any possible error within any function - even when there was nothing what could be done within given context. My "careful" coding became never ending effort.
With the Let it burn approach the relieve was enormous - similar to when you throw away a backpack full of responsibilities you shall not bear. Suddenly the coding became a fun run.
Thanks for explaining this.
Fail fast! Python is particularly well suited for this approach that I employ as well.
The real enabler is the "last chance" logger feature.
In reality, robust error handling is an art form where skill only really improves with experience.
Great topic! You covered it well.
YES YES YES!!!!!!!
For the life of me I can't understand why people write so many useless try-except blocks in every function that just don't help anything and make the code worse... and often also swallow exceptions they really shouldn't...
I can guess this video idea came from doing consulting work and seeing such code...
The only time try-except blocks make much sense to me is when the error is an error that could happen even if I know my code is bug free (for example, a resource missing error).
It really depends on what those try catch are used for. Try catch for logging and printing is just retarded but to run logic on failure to handle stuff gracefully is the way to go.
@@andip3domi702 Yep.
I like way how GoLang mages errors. That way you have full control of each error.
BTW liked this video.
Fail fast is important with few caveats in mind as you said. I would insist in "Exception are for exceptional behaviour", sometimes handling functionally the issue when it is business related is a better idea. Code has clear typing modeling what can or cannot exist, and conditional are way faster than exception.
it's a good approach for some cases imo i think no one can escape errors or predict them and also for more complicated code you would end up with a bunch of ugly "if" statements and we back to zero, but I like the idea of making the obvious errors to not be an error and make the process move on it's brilliant, we don't have to raise errors if we can handle it properly.
Nice advice! I follow the same practice when possible. Even still, when errors become visible, you should not start shooting try/catch everywhere, you catch the error, at a point where you can do a proper handling that makes sense at that point of the code, according to the status of the project, to what user's expect, to what you want users to do in that case. Don't make you're code unreadable with insane try/catches specially at points where you don't even even know what to do with the errors.
excellent video. if someone’s not convinced about fail-fast after watching this, they’re a lost cause.
Literally had this case today at work, though kinda the opposite...!
I need to apply a database update for a bunch of items, but one of them was failing. I had left all the error handling to the outermost layer, such that the error was causing the whole database tramsaction to fail.
Easy to therefore find and diagnose, but i realised i really wanted the error handling to allow other items to pass still.
By having the "let it fail" principle, i could therefore choose the point at which i was prepared to handle the exception because i didn't want to fail at that point, but to log and then proceed with the next item.
It makes for much more sensible error handling at the point where the error could be properly processed, not constantly logging it until someone knows what to do with it.
Other parts of legacy code does do that 'print and raise' handling, and it becomes hard to actually work out what errors I'm actually trying to handle and to get the severity/regularity of them
In the Web API context, a favorable approach is to avoid exceptions in the sense that is best to return a "union" Result type that may contain either the correct value or the error that happened. Exceptions might slow down considerably the calls depending on volume of calls.
What if an actual exception occurs that wasn't raised by the user? For example, if we're unable to connect to the database
Thank you for your respond
@@dynamicgrad3820Errors as Values are without a doubt, better. It's alright to use exceptions in python for building little automation scripts; but for web APIs and stuff, languages like Go and Rust win with explicit errors as values. You can't ignore the error implicitly there, you choose either to crash or to handle (or maybe ignore sometimes).
There are no exceptions in Go, if you try to connect to a database, there are 2 values returned and you need to check if the error value is there or not. Rust did it better imo.
@@dynamicgrad3820 there is always a high-level exception handler as a middleware for unpredictable cases. This cannot be omitted obviously. But most exceptions such as when an entitity is not found, are not "real" exceptions.
It's a good approach and appreciate you sharing when it should not be used!
"Dead programs tell no lies". ("The Pragmatic Programmer: your journey to mastery" by Andrew Hunt and David Thomas).
I agree with this philosophy. Often there is a mantra that the program "must not crash". Except at the very top level, there are often no
meaningful recovery actions, dump the transaction (or run of the program) and start afresh.
Otherwise the implementor, via their code, has 'lost control of the application's internal state' and all results are then suspect.
Thanks for your insights and this video.
Great to hear a bit of sense on the matter. Similar to seeing junior engineers using dict.get, i ask why and they say 'incase the key isnt there'. Great so now we just get a type error 2 lines down when you try and add an int to None
im confused, if using dict.get, wouldn't you do a check right after to make sure the result isn't None? or i guess it might be better to let it error out in order to fix the code to ensure the dict always contains that key.
This is some valuable content, thank you Arjan.
I love how amazingly you handle any topic! Thanks for your videos!
You’re welcome - I’m happy you like the content ☺️.
I fail fast and loud while also tolerating unexpected things from system integration, log and raise.
Another good way to design your own architecture is to return empty native types. e.g. if we were expecting a populated dictionary back we can try: / except : return {}
callers or components can then still check using "if not"
LOVE error handling videos!
Great explanation, as always ☺☺
My colleagues' favourite technique is to catch-log-throw at every level all the way up. When there's eleven logs, you won't miss it!
Have you ever read Joe Armstrong's dissertation on this subject? Amazing stuff
Yes, I use this strategy, but did not know it is called Fail Fast :) Anyways, learnt a great deal of error handling.
I deal mainly with program automation, where the users don't know much about how the scripts work, and can't provide good feeback, and will simply stop using them instead of letting me know. The scripts are also relatively small, so logging is not practical. I've made it standard to include quick tests i.e. if array.len== 0 return None or if mxs.superclassof(selection) !=geometryclass return None. Any returned None than triggers a pop-up t alter the user to the source of the problem. I apply the same structure in program specific scripts (maxscript) for a consistent behavior and users can actually let me know why it failed, so it can get fixed.
This may be fine when you yourself are the user of your software, like your backend example, but a lot of software doesn't have the same luxuries. An end-user app that adopts "fail fast" usually leads to "uninstall fast" and lost business forever.
Say that to Nasa which has sattelites running for over 30 years and every code base in Nasa is made with fail-fast approach. They even released a paper about it
Loved this video, thanks!
Glad you liked it Hector!
I need this video, perfect timing 💪🏾
Glad to be of help ☺️
In case of key errors I prefer to create data structures like dataclasses and manage errors at instantiation level usign invariants or a similar mechanism.
Did you try the exception groups that came with PEP 654? Combining with your tips number 3 this is powerful.
Some people at my company tend to think they need to silently catch exceptions and return null/None. This makes us end up with code that swallows exceptions and it takes a lot more debugging to find out where the issue is than if they had just let it crash and burn!
What would you do when you need to search for several files and want to inform the user that the 2nd, 4th, and 5th files were not found? If you raise an exception, the program will stop running when it doesn't find the second file. I'm a beginner programmer and I'm looking for a solution
@@dynamicgrad3820One thing you can do in this case is use the try catch blocks to catch the error and if this error occurs you can append the file along with any useful information about the file in a list, kind of like a dead letter queue. Try to be specific with the particular exception you are catching, you want other error types to crash and burn.
Try catches do have their uses, we just shouldn't use them without any good justification for it.
Thought, if its just searching for a file in some directory, there's probably a way of searching and returning a bool, if the file exists or not, rather than an error.
@@dynamicgrad3820 Letting it crash and burn is for when the program is in a state where it never was supposed to be and was never intended by the programmer. It means the code is broken. This is different from a user error. If it expected in your case that certain files could not exist, you can return some kind of result object instead of raising an exception. And let the user know what they can do to solve the issue.
@@dynamicgrad3820 sometimes it's better to just have an if statement and do like:
if os.path.exists(path):
do something
else:
log into a list
, then output to the user that XXX files were not found if list is not empty. Yeah, you could do the same thing with a try/except block as well, but something like that i'd end up just using a if/else statement.
edit: unless you don't know the directory path of the files up front to begin with and need to do something like a os.walk() to iterate through directories/subdirectories to find a file, then if it returns None, log it into a list and output to the user it couldn't find it.
I've had to do this kind of searching for files with numerous programs.
@@dynamicgrad3820 This is very trivial - you can use chatGPT or any LLM. It helps a lot if you have a simple issue.
files_to_find = set(...)
files = set(os.listdir("some_directory"))
missing = files_to_find - files
if mssing:
raise FileNotFoundError(f"missing files: {missing}")
You can do it with a for loop or ith list comprehension too. Although I'd focus more on tutorials, leetcode, just to get the basics.
I don't really agree with this video (apart from the caveat section which seems is at odds. As the owner of a company developing an iPaaS platform providing pretty heavy weight integration with ERP, Payment, eCommerce, and whatever systems, you really want to augment the errors so users are informed what went wrong and where - in sensible language they & understand. Not only does it give your users an idea of the error, the cause and maybe a hint as to the resolution, but it also gives your support and development team. You probably want to retry transient errors (such as retrying webservice calls). Do you want to fail fast when you can wait 500ms and retry the operation where it will likely succeed? And if need to fail, you want to fail gracefully with as much info to your logs and users as possible.
Yep, I do. Though I want to move to railway-oriented programming.
12:48 Using "except Exception" will not prevent syntax errors, as the interpreter identifies those before the code even runs. It would catch NameErrors, but those are trivially recognised statically by any linter, so it's not a big deal. It's still a decent way to write a catch-all at the top of the program. On the other hand, one should _never_ use a bare `except:` as that would catch many wrong things (e.g. it would catch Ctrl-C and prevent you from stopping your program, which you almost never want).
You forgot a 3rd option. You can create a decorator function for handling all exceptions and then just add the decorator to whichever function(s) you want. That way you don' t need to encapsulate code with try/except blocks everywhere and keeps the code clean. You can have the decorator returns things like the traceback, the function name where the failure happened and the exception. For Example:
def handle_exceptions(function):
"""
wrapper/decorator for handling exceptions and returning the traceback of what caused the error
and return value(s) of the function
"""
@wraps(function)
def wrapper(*args, **kwargs) -> tuple:
try:
result_values = function(*args, **kwargs)
return (None,) + result_values
except BaseException as e:
# extract traceback details
tb_str = traceback.format_exception(etype=type(e), value=e, tb=e.__traceback__)
error_info = ''.join(tb_str)
return error_info,
return wrapper
Then just add @handle_exceptions decorator to whichever function you want.
This is bad...
Firstly, you don't want to catch BaseException.
Secondly, it may cause your code run longer than it should, causing a different error. Like a whole video is about it.
Thirdly, it's easy to forget what exactly this "handle_exceptions" does and use it on a function that returns nothing, or worse, in some case returns nothing.
Handling exceptions on top level is way better.
@@Plajerity this is not bad and there is nothing wrong with catching base exception under certain circumstances. It was just an example you don't have to use base exception. I had to use it in my example for some code because an error I was receiving was not being captured by any exceptions except base exception. I don't see how it would be easy to forget what it is based on the name of it. It doesn't have to return none if no errors , it's just an example. Im not sure what you are going on about with it making code to take longer, because it doesn't. Also, you can't always handle exceptions at top level, especially if you are dealing with multiple processes or threads.
@@resresres1 Maybe under certain conditions you can catch BaseException, but I definitely woudn't make it as decorator. Just one-time use. Ideally you shouldn't use it.
I didn't mention you modify the function output, making it more complicated.
You're right about multi-threaded workload, sometimes it's an issue. I have too little experience to say anything constructive... In such cases I prefered to return an exception and catch it as soon as possible, checking if it's an exception instance.
Of cause you can also just chain the exceptions you want to handle and then raise a generic exception for every thing else. You can do that in a single try except block.
The only reason i ever use the try .... 'error {e} occured' is if i have some script that runs perfectly fine for 99% of the time and i need it to continue after some exotic exception . I have automated a browser game and i need the script to run for long periods of time for example . Only at the top level and probably a temporary solution.
I'm currently using assert statements a lot.
Nobody: ......
UA-cam Coding Channel: "I debug by letting my code crash and burn as soon as possible"
Crowdstrike Engineers: 👀😐
Fatigued future crowdstrike engineers who watched this video: "I debug by letting my code crash and burn as soon as possible"
World war IV starts the following day 💀
A good film, but I’m wondering about a situation where a function should provide more than one piece of information about an error. For example, if I'm looking for three files, I would like to inform the user which of those three files could not be found. In that case, using try-except at the highest level is not able to catch all the errors.
12:17 Bare except clauses are problematic, but this is not one of them. A bare except clause states no exception class at all. This catches even SystemExit and KeyboardInterrupt, which is why it’s so problematic. This is, however, “catching to general of an exception,” which is a problem for the reasons discussed.
At 10:46, "You gotta do what you gotta do" - Futurama?
nice one
could you review datetime library next time please? or at least put it into your randomizer
how can I get that intellisense in VSC? 🙏 Thank you
Why not using an error handling decorator and log the error along with given function parameters?
oh damn, i just typed this but didn't see you already had mentioned it. I have an exceptions decorator for this exact thing that gives me the traceback, function and the error.
Elixir | Erlang
Imho java-style exceptions are a failed experiment. Rust-style errors- are- return- values works well in practice, but only because there's syntax and macros to hide the plumbing. Pannicking is a last resort, as in a many thread or many process application it leaves you in a what the hell state.
I hate so much try/except blocks, it is so unnatural.
God I hate those `except` blocks saying `print(an error happend .... {err}); raise`; what's the point? Just remove it and, as you mentioned, make the code much easier to follow
This way you can standardize your error messages and then parse your logs easier for example.
Yeah, that could be one use case I guess
@@tuobgWhy would you need to parse your logs easier? When I parsed some big files I knew ahead what values I'm looking for, pretty easy to find lines with grep. You could use a custom error I guess. I don't get it how standarizing error mesages could help.
I just want to know why your error messages aren’t in Dutch!?! 😂
Joking aside, great content as always!
chatgpt code tends to do that type of error handling you showed near the beggign, it does nothing, makes code harder to maintain and gives you less information
python has ruined devs to think that not handling runtime errors is okay, hope yall never touch production code
Fail Fast. Fail Hard. Fail Often.
Yeah, raising a value error for a country code? That’s a custom exception.
Rust forces you to do that
When I was a child and learnt that the ocean was coming to the Netherlands, I was shocked and couldn't sleep for fear that the Dutch would be flooded and all die. I still worry about that to this day.
P.S. I'm glad you survived, Arjan.
How many minutes have we lost in these window? I think Nelson will produce this year. Not sure an old Sterling will win us the title.
Another way for guard clause (if you are ok with AssertationError) _assert my_key in my_dict, f"Missing {my_key} in my dict"_
Assert statements are ignored in compiled code. They shouldn't be used for guard clauses in production code, they should only be used during development.