Some things in the talk I found interesting: Sometimes I listen to things like "make sure your tests are deterministic" and "run your tests on every commit" and thing to myself "who doesn't write tests and run them all the time now?" Then I have to remind myself how a big a shift it was for me to start doing this. Sometimes we need to hear the obvious. I like his point on code coverage. Global code coverage goals are trash, but knowing your coverage on any given commit can help you catch unforeseen interactions. 33:46 Two asserts is a dumb reason to break up a test that simple, concise tests be damned. Things that test a single operation (in this case, did the user log in) should be tested in one place where it's easiest to understand why you're running those specific asserts. Not sure about other languages, but java has a couple soft assertion libraries that allow you to run multiple assertions and only fail+print errors all at the end of the test. I'm especially surprised given AssertJ has a SoftAssertions class and he gave the library a shoutout later in the talk. The title of the video needs to explicitly say this is a unit test talk, not a general test talk. Integration and ETE tests will generally not have setup steps that are reasonable to inline per test method, for example. I would be interested in hearing him talk about the differences between good unit and integration tests. 43:15 Who has different departments for integration tests and unit tests??? Am I hearing what he's saying correctly? Are the people writing individual components not also responsible for knowing how that interacts with other components? I get not managing all the interactions, but different departments feels like an organizational bug. Maybe someone who's done something like this can tell me why this might be ok, but right now I'm horrified.
Regarding 33:46 : Junit 5 has assertAll which is compatible with AssertJ or whatever assertion library you prefer, and Junit 4 has a less-so-nice ErrorCollector.
@@SirWolf2018 Thank you, didn't realize JUnit4 had this available. May have to tweak some tests at work :) I'm still surprised this feature didn't make the talk.
I agree that 100% code coverage doesn't mean your code is bug-free, but 80% coverage means that 20% aren't tested at all. So 100% is a precondition, right?
I really don't think one strive for 100% coverage. In most cases that just is not practical. E.g., if you are writing in C or C++ you cannot test the main function because it is not possible to call main from another function. In cases where unit tests are sufficient to test the application you cannot test anything that directly communicates with the outside world. Everything that is replaced by a mock is not tested unless you also have tests where that thing is not replaced. One should certainly strive to test as much application code as possible but 100% is generally not something that one should strive for most of the time because at some point it is no longer worth the effort. It is also not entirely clear to me what you mean by '100% is a precondition'. A precondition to what? To bug free code? Testing gives more confidence but never certainty. If you want truly bug free code one should look into automated theorem proving.
@@chrisdams "Everything that is replaced by a mock is not tested unless you also have tests where that thing is not replaced." Right, you should have both. One where you use mocks and verify the behaviour of your code. Does your code use the external dependency in an expected way? But you also need end to end tests where you make sure you're external dependencies work as expected.
@@timmgutowski8739 It is all very context dependent. A throw-away script may not need testing at all. In some cases unit testing is enough. In some cases one may only want to unit test the complex part of the logic. For rather complex server-client applications one probably needs end-to-end tests. It all depends on how complex the thing one is writing is and how much certainty one wants or needs.
@@chrisdams Sorry, I disagree. In my experience, the approach to leave "simple" code untested leads to code that eventually becomes too complex to be tested. Too many developers miss adding tests when the code becomes more complex. But when tests exist, they tend to expand them. But yes, this is context and team dependent.
@@timmgutowski8739 It is indeed a danger that people do not add tests when it really becomes necessary. That is why one has to watch their colleagues... I do not really believe in code that is 'too complex to be tested'. I have added tests to software at any point in its life cycle. Let me give an example of tests that are not be needed. I once worked on an application that had a very simple GUI. Basically like three input fields and a 'start' button. The process that was started by this was rather complex. In this case it makes lots of sense to only test the complex process behind it and not the GUI. The GUI in fact never became more complex. Well, at some point it also acquired the ability to show a yes/no confirmation question but never more than that. Given that testing GUIs is relatively complicated and often time consuming and that this GUI really never became complex I am 100% sure this was the right choice. Writing tests for the GUI would have been an enormous waste of time.
Some things in the talk I found interesting:
Sometimes I listen to things like "make sure your tests are deterministic" and "run your tests on every commit" and thing to myself "who doesn't write tests and run them all the time now?" Then I have to remind myself how a big a shift it was for me to start doing this. Sometimes we need to hear the obvious.
I like his point on code coverage. Global code coverage goals are trash, but knowing your coverage on any given commit can help you catch unforeseen interactions.
33:46 Two asserts is a dumb reason to break up a test that simple, concise tests be damned. Things that test a single operation (in this case, did the user log in) should be tested in one place where it's easiest to understand why you're running those specific asserts. Not sure about other languages, but java has a couple soft assertion libraries that allow you to run multiple assertions and only fail+print errors all at the end of the test. I'm especially surprised given AssertJ has a SoftAssertions class and he gave the library a shoutout later in the talk.
The title of the video needs to explicitly say this is a unit test talk, not a general test talk. Integration and ETE tests will generally not have setup steps that are reasonable to inline per test method, for example. I would be interested in hearing him talk about the differences between good unit and integration tests.
43:15 Who has different departments for integration tests and unit tests??? Am I hearing what he's saying correctly? Are the people writing individual components not also responsible for knowing how that interacts with other components? I get not managing all the interactions, but different departments feels like an organizational bug. Maybe someone who's done something like this can tell me why this might be ok, but right now I'm horrified.
Regarding 33:46 : Junit 5 has assertAll which is compatible with AssertJ or whatever assertion library you prefer, and Junit 4 has a less-so-nice ErrorCollector.
@@SirWolf2018 Thank you, didn't realize JUnit4 had this available. May have to tweak some tests at work :) I'm still surprised this feature didn't make the talk.
You're welcome.
use mutation tests instead of coverage to get an evaluation of your test quality
Thanks Jeroen! Really great knowledge here. Most of the principles and techniques shared are applicable to other languages like JavaScript/TypeScript.
I agree that 100% code coverage doesn't mean your code is bug-free, but 80% coverage means that 20% aren't tested at all. So 100% is a precondition, right?
I really don't think one strive for 100% coverage. In most cases that just is not practical. E.g., if you are writing in C or C++ you cannot test the main function because it is not possible to call main from another function. In cases where unit tests are sufficient to test the application you cannot test anything that directly communicates with the outside world. Everything that is replaced by a mock is not tested unless you also have tests where that thing is not replaced. One should certainly strive to test as much application code as possible but 100% is generally not something that one should strive for most of the time because at some point it is no longer worth the effort.
It is also not entirely clear to me what you mean by '100% is a precondition'. A precondition to what? To bug free code? Testing gives more confidence but never certainty. If you want truly bug free code one should look into automated theorem proving.
@@chrisdams
"Everything that is replaced by a mock is not tested unless you also have tests where that thing is not replaced."
Right, you should have both. One where you use mocks and verify the behaviour of your code. Does your code use the external dependency in an expected way? But you also need end to end tests where you make sure you're external dependencies work as expected.
@@timmgutowski8739 It is all very context dependent. A throw-away script may not need testing at all. In some cases unit testing is enough. In some cases one may only want to unit test the complex part of the logic. For rather complex server-client applications one probably needs end-to-end tests. It all depends on how complex the thing one is writing is and how much certainty one wants or needs.
@@chrisdams Sorry, I disagree. In my experience, the approach to leave "simple" code untested leads to code that eventually becomes too complex to be tested. Too many developers miss adding tests when the code becomes more complex. But when tests exist, they tend to expand them.
But yes, this is context and team dependent.
@@timmgutowski8739 It is indeed a danger that people do not add tests when it really becomes necessary. That is why one has to watch their colleagues... I do not really believe in code that is 'too complex to be tested'. I have added tests to software at any point in its life cycle.
Let me give an example of tests that are not be needed. I once worked on an application that had a very simple GUI. Basically like three input fields and a 'start' button. The process that was started by this was rather complex. In this case it makes lots of sense to only test the complex process behind it and not the GUI. The GUI in fact never became more complex. Well, at some point it also acquired the ability to show a yes/no confirmation question but never more than that. Given that testing GUIs is relatively complicated and often time consuming and that this GUI really never became complex I am 100% sure this was the right choice. Writing tests for the GUI would have been an enormous waste of time.
Yeah, re-running CI to make flaky tests pass is quite annoying
Exelent