Why I Don’t Unit Test
Вставка
- Опубліковано 26 жов 2024
- Wellp. Y'all knew this was coming. Watch live on Twitch / theo
Keywords: TESTING E2E UNIT TESTS CI CD END TO END INTEGRATION TESTING REACT JAVASCRIPT TYPESCRIPT
LIVE EVERY WEDNESDAY AT ROUGHLY 1PM PT
Twitch: / theo
Twitter: / t3dotgg
Instagram: / fakiebigfoot
Everywhere else: t3.gg/links
And obv shoutout Idez as always 🙏
FULL VOD: • STREAM VOD 7-24 - Easy...
I actually agree with most of your observations. I don't fully agree with your conclusion though.
I have a similar but different view of unit tests: as a development tool. I write unit tests as part of a process, to verify my own assumptions - will this work, will the API be appropriate in client code, and so on. This isn't extra work for me - I can code something faster by verifying each step of the way, not fumbling my way through the dark. I work faster and produce better work with tests.
I usually check them into source control as well. But I am always ready to throw them away, at a moments notice - and I make a point of telling everyone, if it gets in the way of making a change, throw it away.
I think, If that's your position, unit tests are perfectly harmless - they can provide value as a development tool, and as documentation, for as long as they're still viable. As long as you don't view them as a safety net, you can safely throw them away as soon as they no longer provide value. If that's after writing the client code, before even checking them in, or once you have E2E or integration tests covering the important parts, great, toss them. But they can still be part of an effective development process.
I like this take and would probably be convinced if you were an eng at my company arguing in favor of unit testing
I'm still thinking about this take - very good
I think the important thing here is the willingness to throw them away. One of the biggest criticisms is the "overly restrictive lattice" an overblown test suite becomes...if it's only viewed as a means unto an end and the eng culture is fine with throwing them away...that kills a LOT of the inherent footguns in unit testing practices.
"will this work, will the API be appropriate in client code"
I want to underscore this point as well. I can't tell you how many times I have written a unit test for a thing I wrote that will be called by others only to realize, "Yeah this API or function prototype is clunky". I end up simplifying it and thus the client code is simpler too.
Came to say almost the exact same thing. I would rephrase the problem as "people build a lot of unnecessary unit tests to meet arbitrary coverage metrics" which is super valid. I've seen FE unit tests for content (meaning if someone changed a word it broke).
But for the FE, I follow Kent C Dodds unit testing strategy (react testing library)...I write them to mirror the requirements. If I click on X on the screen, Y should happen. If it doesn't, that means either the dev broke something and we caught it or the requirements changed, thus the unit test should change. Usually if your pages aren't overly busy, that means you're testing 1-5ish things per page and your code coverage will be prob 50-60% at most (and probably not even that high).
And to Rasmus's point...when the test passes I know my unit of work is done.
i agree when it comes to UI stuff, but for example if you write a parser unit test are super usefull because the feedback loop for changes is much shorter
This! In general, for non-trivial algorithms there is often the problem that if you change one small part to fix that one bug, another bug will be introduced. Without unit tests, you would not even notice something broke until it does for some obscure edge-case (which you had to do by hand). I honestly would be unable to develop any complex (in the sense of non-trivial algorithms) program without unit tests. Maybe someone smarter doesn't need them because they can visualize every possible code path in their head but I definitely can't.
This is a really great example of unit tests being the right choice.
Funny, parsers and compilers are one of those cases where I usually assume end-to-end tests are better than unit tests. The whole thing is one big function from input to output so no need to break it down any further. Maybe test going into and out of the IR if one is part of the spec
@@andsnpl theoretically yes but it is very hard to reason at that level and you never know what level actually caused the error you are seeing (if you are actually able to predict the output in the first place)
@@simitron1 true, there are programs for which “correct” is a nebulous concept. I would argue that compilers are not such a case (you write a program that exercises feature X, you compile and run the program, and it either works or it doesn’t). Parsers are even easier.
But if you do have such a program, how would you define “correct” for a unit test? This is a general problem for tests of non-user visible behavior that gets worse the messier your domain actually is. I’ve had test suites around some software that dealt with US taxes, and the unit tests were always kept green, but that gave us absolutely zero confidence that the system worked as expected
The feedback I give my team on any test is
1. Add unit tests to make it faster to write complex functions (like doing some algorithm).
2. Add unit tests when the cost of something breaking is very high. (In our case, our APIs have 99.99% uptime expectations from our integrators).
3. Add unit tests around code that is difficult to understand for whatever reason.
Otherwise, prefer higher level tests.
Also, tests are not free, both the build, maintain and run. Consider the value of the test before writing one. We still end up with around 50% branching coverage because of being in a regulated industry, but it's not all or nothing.
Only thing I disagree on is point 3, and not because "it shouldn't be tested" so much as "try to kill the code that's hard to understand". There's definitely some things that are necessarily complex, but MOST things don't have to be and I'd rather put the effort into simplifying things
@@t3dotgg ye I constantly see over-engineered rubbish code from unit test advocates.
@Michael NOTHING ever needs to be so complex that you need tests to safe guard you, and if you do then that's just bad/junior engineering and not true SoC.
@@t3dotgg so you do agree on unit testing? 🤗😂
@@t3dotgg There're definitely inherently complex things with unobvious caveats and weird edge cases. When I catch myself thinking either "I don't have enough brainpower to cover all the edge cases mentally" or "it took me 2 days to fully understand it, so I can't trust someone not to break it", it's usually a good time to write a unit tests. Rarely happens though, as refactoring the hell out of a too-complex-to-maintain piece of code is almost always enough.
Agree with not testing UI stuff, it's just a hassle. But for utilities/helpers/services/(or even some react hooks)/anything similar, I beg to differ. Just the thought of having a bug solved before appear again since it was not handled by a unit test is just a pain in the butt.
Saw your discussion with Prime about testing though, and I really enjoyed that! Both of you really made valid points.
Totally agree!
The same way Theo said that there's a big class of problems that TS alone covers, unit/integ testing helper modules, client, services, parsers, etc, makes the UI much more stable and, in the long run, will reduce the amount of silly bugs reaching prod.
The cost to write a test for a parser is so low, considering that a broken parser can break the whole UI, so in this case, it makes total sense.
Even worse is when another dev takes a look at your code. Tests allows you to communicate how a particular unit works.
Check out material by Kent C. Dodds. He's totally changed my perspective on UI testing whether it be React, Vue, or vanilla.
What's a React hook that isn't related to "UI stuff"?
I would argue the UI testing is the most important and if done right it is the best way to make sure your requirements are met. As others have said, check out Kent C. Dodds React Testing Library stuff.
I’ve been doing TDD now for a year and can honestly say that it helped our team deploy quicker and with near zero defects.
Part of TDD is trying to break your code, so naturally you will catch loads of edge cases.
Writing tests after the fact is not very productive, but the safety is useful for refactoring to ensue things still work
Same here! TDD ends up with great code. The important thing is to take care of the business logic and make you code clear, easy to change,.....
Also, writing unit test or TDD is not only a way to test, it's a process of thinking to write better code as well. Our team can write unit test very fast after been struggle for a while on if we want to unit test. Now the time allocation we write test just like, I guess, 5%~10% of production time. The funny thing is now each ready PR must come with test, BUT, we didn't notice any noticible longer working time compare the pr without unit test at all. AND we notice that we have less bug and more SOLID unit for sure. I think it is way quicker to capture issues than thru e2e layer. ua-cam.com/video/VDfX44fZoMc/v-deo.html This video actully helps me to understand we may need put more effort on unit tests.
I've tried test-driven development as well. And if done right, it helps you to structure your program better and to implement it faster, without spending days debugging or having customers yell at you because of the bugs. That said, it probably doesn't make much sense for GUIs, better for logic and backend.
@@MS-ni3uz Holyyy such a good talk, thanks for sharing it, man
@@MS-ni3uz that video is legendary, it very much improved the tests on our team
The real value in tests is to make you think about how your code is designed. If your code can be unit tested, it's far more likely that it's designed in a way that is orthogonal and flexible. This means your whole app is easier to add features to and respond to changing needs of the users.
If you just see unit tests as a roadblock to production, then you're missing the point that the unit tests are telling you... that your code is designed poorly and is brittle. It might be cheaper to deploy a bug and then fix the bug than it would be to write a unit test... but when writing the unit test ensures the function is designed well, it's far less likely to have to be operated on.
When you're not designing functions to be tested, you're going to write brittle functions, with internal state and uncontrolled side-effects, and you're going to spend a LOT of time fixing production... and the cost adds up, and it is accumulated in an environment that has an opportunity cost because it impacts your users and your brand.
Your whole take on making a unit test work to cement the code is exactly what I'm talking about. That just shows the code was written without regard to how it would be unit tested. A unit of code that can't be unit tested is likely to be just as brittle and difficult to use by the other bits of code in your application. The fact that you see unit tests as conforming to code instead the other way around just highlights how you fail to see the real purpose of unit tests to begin with.
This line of thinking is selfish and short-sighted, and irresponsible.
Type-checking and unit-testing are both ways of proving behavioral properties about your code's programming interfaces. When seen through this lens a lot of these arguments fall apart:
"Typescript is just a way of slowing down engineers"
"Catch/fix type bugs quickly in production - safety nets, not guard rails"
"Type annotations just unnecessarily cement the JS data types in place and make them difficult to change later"
Look, there's a very real possibility that the code Theo works on doesn't benefit from unit tests. Just like some JavaScript applications probably don't benefit from taking the time to setup and maintain static type safety. But in my opinion the way these half-truth arguments and vague examples are phrased is just gonna enable a lot of people struggling with understanding automated testing to sink back into their comfort zone, without really elucidating why you would or would not want unit tests in a given scenario.
Theo is a big advocate of typescript, and I didn’t hear any of the other two arguments you are referencing in this video.
@@TimeConvolution That's my point. The same arguments he's using against something he opposes (unit testing) can be applied against something he supports (typescript). They're weak arguments. Both typescript and unit testing are valuable tools for reducing risk and improving maintainability of a product.
you are comparing two different things. Typescript was created for languages like javascript that didn't have type management enabled. A unit test, on the other hand, has only two purposes. The data to be transmitted and the data to be returned. But in order to do this thing, you literally create a lot of junk.
Your comment says nothing of substance. Yes, the point of a comparison is to compare two different things. Otherwise it would be an equivalency, not a comparison. ???
@@MachineYearning is pretty sad they are too obtuse to understand. I hope they are not engineers lol
Static checks create compile time issues for logical errors that would otherwise be runtime errors. Runtime errors must be picked up by tests or by your users lol.
Unit tests also help new developers know they haven't broken functionality off features they don't know about. They act as documentation of use-cases if they are written well.
You can debug them in order to understand control flow and they should document how to use the API.
Like you said these are very weak arguments.
For me unit tests are up to date documentation. And they provide me the safety net to refactor code at any time without taking into consideration tens of thousands lines of code.
but then just write automated integration tests, at the end of the day users are not going to see what happens when you press of this button, they dont care about the button, what they care about is the overarching feature i.e. submitting login form so I think best trade off for FE project is to have automated integration tests and nothing else because that will give you good enough support less tests to write and it will actually work with deployed BE rather than FE devs mocking api calls (which wont help if someone on BE side messes up)
The code itself is the ultimate documentation.
@@eplugplay8409 Apparently reading code is too hard when you are bringing in cheap teams from overseas.
@@nishantdesai3705I feel like this take is biased to the frontend use case. If you are writing any backend service that has reused components you need some tests specifically for any data writing service, less you want to implement a whole rollback system with God knows how many check points $$$. Which of course, will itself have no tests 😂
@@randEveScrub yup, very true, on BE I can see that you dont have any other alternative, so yeah I agree my take is more for frontend development
I don't think unit tests are designed to slow engineers down. I think the vast majority of engineers don't understand how to write good tests. Partially that's because writing good tests is difficult, but mostly because the word "unit" makes engineer think "test the smallest possible part of code". Usually that ends up being a function or a component, which really shouldn't be the target of a test. If you are testing features and not implementation details, tests can be really valuable when refactoring, but again the hard part is defining logical tests. I agree 100% that code coverage is a poor metric though.
Yeah, that's how my college told me on how to do unit test. After developing backend app, i realized that i only need to do unit test on some part of my code that I'm gonna change or add, like payment method, data source etc
Unit tests done right makes you faster.
Yeah, unit tests are designed to actually speed you up. Maybe not when coding new logic, but certainly when modifying existing pieces of the app
@@Titere05 even in coding new logic. You need to test it anyway. You can write test > code > test > code over and over again before having to run e2e or even manually checking things.
The only cumbersome part is the fixture part. But running the system under test, verifying it, and even cleanup are all pretty easy to write.
The problem is that very few knows practices and design patterns with tests. And we just inherit a senior’s noob skill testing (your senior can be a great dev without having to learn how to properly tests) and that noob skill testing gets passed down over and over again.
Just imagine, how many practices and design patterns we know in writing code. And how little do most people know about actual testing practices and patterns.
Note: you dont need a lot of info, you just need to be familiar with the test org structures, phases, common design patterns. Everything could probably be learned in less than an hour. But most will never body learn those because most dont even know they exist. So we revert back to procedural programming with our tests 😅
THIS. We have a junior dev today actually break several tests and when I asked them to open the site, it was visibly broken. Once they fixed the code they wrote, the tests passed again. The original functionality was back in place. They didn't have to modify my tests at all. I'm very careful about *not* writing tests that cover implementation details, but most people I see writing tests are mocking and spying all the things. Those are the tests I wind up deleting like Theo's FB engineer. Good tests provide a lot of value and stay untouched unless the input/output changes.
03:45: Fair enough, but what about bugs that don't get reported? What about users that are negatively impacted by the bug, notice it, then leave and don't come back? I feel like the latter point is especially relevant when your brand is not well known to the public and lacks the "street credit" to afford having user-facing bugs in prod.
Automated tests (unit, integration, and e2e) are tools to help ensure your code is working as expected and to help catch possible regressions as you make future changes. Just because a tool exists doesn't mean you have to use it, but I would like to point out that the "be able to quickly patch bugs in production" workflow doesn't work for industries such as aerospace, medical devices, etc. where reliability of the software is crucial. If you're not working in these types of industries then you can probably get away with what Theo is suggesting in this video.
yes you need to use it. because the company in were you will work will tell you that just use unit test, when you are programmer you don't choose anything even the framework , you just get hired and start working with the stuff the the company uses, if they use express and you don't like it, good luck trying to convince your CTO. I see unit testing just like a check box to make the pipeline compile and no more, have you ever tried to read the unit tests of other programmer that was fired 1 year ago ?
Hi Theo, I appreciate we must have very different clientele as mine would not tolerate having faults discovered in the production system by end users. I could not disagree more with your position on unit testing and think making such blanket assertions could be regarded as irresponsible.
The increasing reliance society has on software means when it goes wrong it might not just be a case of saying "oops, sorry we will go away and fix it" and could have significant personal consequences.
I am not talking just about self-driving vehicles or rocket control systems, which obviously have potentially disastrous consequences should there be a failure. I am referring to more mundane systems such as applying for emergency social security payments or reordering vital medication supplies.
As an industry we have a vital responsibility to ensure our code is as robust and fault-free as possible. Establishing a mechanism to trap faults early is an essential part of the software engineering process.
AMEN!
Agreed, this is a reckless (and possibly incredibly myopic) post. There's a plethora of systems (ordering, ERP, Payment, distribution, db, infrastructure, medical, aviation, control...the list goes on) requiring some level of formal testing before deploy to release. There are a swathe of organisations and indeed scenarios that either do not have capability to monitor production, cannot monitor in production or where the consequences of a critical fault in production would be disasterous.
Completely agree. My customers would not put up with the system breaking every other release. It's all fine and dandy if you can crank out a build in 7 minutes, but how long was that bug out in the field causing havoc with your customers before it was discovered? How many of your customers had to suffer before one of them decided to contact support about it?
Using facebook as an example doesn't inspire me with confidence. So what if FB shows me the wrong post or doesn't show me a friends post the minute they posted it. But, as you say gilmoretj, the vast majority of software systems out there have some real consequences for bugs getting into production.
Well said. He is an amateur and just wanting more views on his videos.
Completely agree Sr!
I see unit tests as a means to defend a code against the future self who could modify a function in such a way that breaks other functions that use this modified function, but I also think the code coverage should not be enforced. Much more important are still integration tests which should detect any errors that dev could do. I personally believe that if you are working on MVP, the last thing you focus on are unit tests. When working on MVP there is high possibility that code in many parts of an app is going to change and making old unit tests useless
I think it would be more helpful to explain what safety nets you employ and how you design/create them. I think a lot of people see TDD as a safety net.
Cheap as possible to fail... sounds like a job for a unit test or integration test.
Tests can be used as a form of documentation.
Unit tests should test inputs and outputs of methods.
100% code coverage is definitely not needed. 80% is a good start.
I agree that typescript can help you find issues due to compiling.
I always found that engineers who did not know how to test or code for testing were more likely to find reasons to avoid them
I mean what do you expect from someone who looks like he is in his early 20s who's trying to act as an expert guru
thats the issue with unit tests tho. they are supposed to be guard rails but they fail at even that when you can just mock the implementaion wrong anyways. so its not like its saving you from people that dont know how to write tests anyways. you can even expect a certain implementation thats not correct. its just a big waste of time, you are gonna make mistakes might as well free dev time to build better code as opposed to getting code coverage
@@zapfska7390 Unit testing is a skill that takes experience to do well (a skill that is lacking in most developers and most teams). What they do for you is alert you when something breaks. The only alternative to that is manual testing (by manual test teams, or by the end users).
@@vaxrvaxr actually the alternative is E2E tests which are actually more robust than unit tests.
@@juleswinnfield9931 But unit tests makes it far clearer where things went wrong.
I create unit tests for both Api and UI, but it's important to only test important and worthwhile use cases and not blindly look at code coverage.
I work at a company that is extremely risk intolerent (which is justified) and so we have many layers of protections to prevent bugs going into production, including extensive unit testing. When I started here I was a strong proponent of unit testing (this is my first dev job) but now I am super skeptical. I have seen so many bugs that (fortunately) got caught by other forms testing that had unit tests 'covering' them but the unit test was bad because the dev wrote bad unit tests. This was not really the devs fault as there are so many reasons this occurs including:
- not being able to think of all the possible edge cases
- frustration at all the irrelevent stuff you have to consider to test poorly written legacy code
- tiredness or lack of motivation (because you've spent too much time writing unit tests)
The most egregious example of a bug that had a unit test that I've seen was for an Api call that literally did nothing where the unit test was literally just a test that the method exists
I 100% agree with the view that writing unit cements your code in cement as if you want to rework bad code most of the work comes from fixing the unit tests (especially on legacy code bases).
Your take on make safety nets not guard rails is spot on
fortunately my company doesn't have a code coverage requirement and the main person pushing for code coverage has left.
Ha, maybe we worked at the same place ?
You just described every single commercial project I have ever had the misfortune to work at :)
With FB, the lack of unit testing is because of a *myriad* of other things are blocking and slowing down other engineers, and that FB maintains no other environments besides production. IMO, if the CI/CD process for landing a diff didn’t take days, it might be more possible. My current team deals with unit tests at Meta, and they are the least intrusive and often most informative part of processes. However, they are few, fast, and carefully chosen.
I agree and disagree that unit tests are there solely to block devs. I agree in that unit tests usually end up doing that because they’re overly and poorly implemented. I also agree with you that unit tests need to be viewed through the lens of “does this add value”. Treat unit tests as product code, and when adding them ask if they’re adding value to the product or just a blanket of false security.
I disagree in that when well implemented they can serve as living documentation - not just internally but externally. Also, when composed well and written to be fast, they can help catch issues before they might come up at potentially more painful (and longer running) stages like image/container builds, etc. It’s a lot easier to run a quick slew of fast unit tests in a minute then deploy a fix for a caught issue than wait 20-30 minutes for a service to redeploy every time a bug is reported or interrupts a build. There’s also that, even in types safe or strongly typed code, you can still have logic errors. Don’t get me wrong, typed languages can reduce a *lot* of common errors, but even in individual smaller units of code, it’s still possible to write perfectly typed but logically flawed code.
Finally, writing unit tests can serve as a canary for confusing or convoluted code. If a unit test takes longer than a second or two to execute or the mocking is difficult to implement, it’s a signal to me that the code it’s testing might be overcomplicated/slow itself and I need to look at simplifying it. That brittle feature that has umpteen billion tests? Like you said, that’s a sign the brittleness needs to be fixed, which can be far more difficult to gauge sans unit tests. IMO - if writing unit tests feels like it’s blocking devs or slowing things down, look at *why* implementing those tests is difficult. The why is usually an issue with the implementation, size, or even architecture of that feature that needs to be fixed.
Sum total, you raise a *lot* of really great points Theo, and I deeply enjoy your perspective. See you in the next one!
A bit off-topic but I have no idea how anyone with integrity can still work there after its been exposed as a traitor to humanity. Human dumbness has no limits apparently and that's how they keep getting away with it.
All code is tested eventually. You either choose an approach or one is chosen for you. Users as tests. Monitoring as operational tests. Typescript linting errors are tests. Many things are tests and there can be a mix but ultimately I think tests are just automation. Devs love automation and so the reframing is an easy sell.
Unit tests are supposed to shift left. The Test Pyramid (Martin Fowler) approach came from the backend, it doesn't work as well for the UI because the UI has stuff only a human can know (CSS) and the backend style isn't quite as recognizable as something like react testing library with the DOM (or previous approaches like enzyme). The closest is plain JS functions like helpers. I extract helpers out of components but it's not the same. Sometimes a component doesn't have a unit-y aspect to it. In that case you are looking at RTL, snapshot testing (outdated) or some reach for end-to-end. Feedback loops are everything and some of these tools resist a low latency feedback loop. If you are automating with tests then you should mostly be trying to write unit tests. This can be tricky because of the DOM. Things like Storybook are sort of a similar flow (but with a lot of setup and boilerplate). I don't know. I still try to aim for this Pyramid and I think Cypress/Puppeteer/Selenium are too heavy (which is still the Pyramid).
The problem with learning a weak testing workflow is it doesn't translate to the backend or fullstack frameworks. In backend there is manual testing (aka no testing) and automated testing. Many choose manual testing which seems fast in the beginning but slows down as you build more and more features. More projects grow in complexity so the problem with "I'll just do it live" is that you get more and more regressions. I think it's better to learn or experience a nice testing flowstate at least once.
The problem with saying that tests are "just a tool" is that this statement is weak sapir whorf. It could be that testing helps you check your assumptions and get used to that even if you aren't writing tests. This might help you when you are making production changes. The problem with saying that TDD (not testing) solves everything is that, in practice and measured, it doesn't seem to make any difference at all. Hillel Wayne's talk about Empirical measurement in software: ua-cam.com/video/WELBnE33dpY/v-deo.html
(25:13 is where he starts talking about TDD)
Testing is a forever skill. Basically every language on every type of computer has a unit testing library. There is no paradigm or tool that can fix at least logical errors. So we invented automation for "check a little while I work", this is how programming started. If you find a bug, you fix it and test it so that case is covered. It's about regressions and communication. Imagine any new hot language or framework with any shiny feature ... it will probably have a unit testing library.
We cannot invent a machine or a language that doesn't need functional tests so you might as well learn testing once and have it with you for your entire career. The problem with a small 5 person team is you might not have experienced turnover or brain drain yet. I would not want to join a project where I have to break prod to understand how the app works. However, this sentiment about fixing fast in prod is very similar to Charity Majors' talk against having a staging environment which I agree with (after thinking on it for a while). So in that way, you could have both nice pipelines and automation but also always executing expectations.
But each community has a culture with strong and weak aspects. The tech culture dictates behavior. Between tech cultures, we can't even get our definitions to be the same. So it is very hard to pre-explain a flowstate feeling when we have a language barrier. It's hard to pre-explain a feeling and flowstate to someone even if we didn't have a language barrier. TDD isn't all of testing and there are degrees and gradients even in that one term. Kent Beck explained it in ~2002 in a book (tdd by example) but even I have telephone-gamed it to a distorted meaning. What does TDD really try to do? What is the point? This is where we could start but some threads just end with "in the trash".
05:41 Typescript doesn't solve functional problems. You expect `helloWorld()` to return "Hello World" but it returns "Oh no". Typescript tells you that it's a string which is correct but you don't see the bug until runtime. Tests run in runtime. Types can remove certain kinds of tests but not all of them. Even with Zod "Oh no" is still wrong. That doesn't mean we should give up.
If your team and project doesn't see value, great. If you are tracking it, great. I'm glad you are explaining the context because this pipeline fix it prod flow I think would devastate the junior devs and interns on my project. "Reverted your commit". :( I'd rather have them have a failing local test on their laptop but, yes, there is a time cost in the beginning. Tradeoffs versus manual testing.
Quite a big difference between writing library code, tools etc. compared to end user applications like a web site. Different contexts have different requirements and appropriate methods of ensuring quality.
I just love this guy for one reason. You are just too confident!!!!! And i totally agree with you. We have an application running in production, and we have over 45k active users. And the entire application was designed without a single unit test, (from backend to frontend) infact as the lead engineer i never loved writing test anyways. And we have only attended to 7 minor bugs report from our customers in 4 years.
Unit tests document what and why your code does as well as supporting integration tests. They have a simple input and a simple output. If the stuff in the middle changes, unit tests don't care about that
TDD makes you faster not slower, because you get a feedback loop for the code you write and any new changes you need. I strictly followed tdd and I was about 50 percent faster than when I started getting lazy again and skipping unit tests. This was for an internal tool and not production
If you write good unit tests they won't fail unless the behaviour is changed
Maybe the code in their company is complex and has high dependencies and poor interfaces so testing then a waste lol
I agree. The advantage of TDD for me was thinking more about how to write good API and to keep stuff decoupled. These days I rarely write any tests at all, because once you've learned how to write good API, you no longer need the tests. Sometimes I write them during development and then throw them away afterwards. Keeping a full test suite is garbage.
Unit testing can be valuable if it actually tests the correct unit: A feature. But most test bases out there test classes, i.e an implementation detail, and make refactoring or rewriting an absolute nightmare.
And I agree on most cases against even properly done feature tests. Usually fixing the bug is faster than writing the test and the defect is not problematic enough. You'll get the report, you'll fix it in 10 minutes.. what's the problem? Or you have a different deployment and manual testing before release in place anyway, so unit testing doesn't even save you anything.
I've yet to encounter a code base with mass amounts of unit tests where in an entire year of constantly running the tests a test has actually found a single bug, or defect. Not even one that could have been found easily anyway, I literally mean a single red test that wasn't a false positive of negatively reacting to a refactoring. I have never seen that. Ever.
Can they have value in high value code that has complicated algorithms where manual debugging can be a pain and unit tests can help you pinpoint where things broke? Yes, absolutely. But 100% code coverage is 100% BS and a huge waste of time.
Everybody should practice TDD at some point, if only in a personal project, because it teaches you how to write proper readable, decoupled code with API that makes sense, but under no circumstances should those tests stick around.
I've been in so many projects where deployment and CI was slowed down because running the tests took 30 minutes. And yes that's not what unit tests are supposed to do, they should run in miliseconds, but in practice that crap happens, because juniors exist and managers exist that tell them to cover their code with tests.
In my experience 50% of tests have zero actually meaningful assertions and the other half is constantly producing false positives or slowing down CI.
Unit tests are extremely important in my view. I cant risk small changes breaking an important module. Before I merge I try to make things work perfectly and manually testing everything after every tweak is extrmely slow. The bugs we can get screw customers over such as incorrect analysis results. Customers could get confused or it could even lead them down the wrong path. A mistake in production could cost multimillions
I think unit tests are useful for some cases. Specifically, any time you have a library that simply takes input, processes it in some specific way, and returns a result that should be deterministic based on the input - no filesystem, database or other external factors involved. Ex. a math library. For most other cases, such as UIs or web apps which usually rely on all sorts of external things (API calls, database, etc), and where results change all the time anyway, it's prohibitively hard to test and tests would be fragile and largely useless anyway.
Well.. Unit tests also brings other benefits to engineers like:
- Improvements in code quality bc forces engineers to write more testable and atomic blocks of code
- Documents the code so it tells you what it should do and how to test it later if you gonna make changes
- Let's you make changes without checking ALL the code and then just check if tests are passing to see if something broke (it's faster en effective sometimes)
> Improvements in code quality bc [it changes developer behavior]
That presupposes they weren't doing that to begin with. When is that assumption true and false, respectively?
> Documents the code
I've heard this many times. I've basically never seen it happen. Never by the tests alone. People usually don't write the global business requirements into their tests, only the technical requirements on the one piece being tested.
To test that when you do the thing the result happens, you often have to create a mock user object, stub out a thing, massage the request object, and so on, and the result is that the test gets plastered with incidental complexity, such that you can't read which parts are the requirements are which parts are mere means towards the end of running that one test. And so the thing it's testing gets tangled up with how that thing is being tested, you lose sight of which is which, and the documentation value plummets.
I'd rather have a clear comment explaining what the user expects any day of the week. Or a comment explaining the subtleties of some complex algorithm.
> Let's you make changes without checking ALL the code
A pure function is the only self-contained piece of code. I think I heard that on this channel, maybe via Brian Will. Writing lots of pure functions means you don't have to test them when you don't change them. A good type system and/or a good system architecture will ensure purity (in the places where it makes sense).
Clear communication, good system architecture and responsible developers bring many of the benefits which are claimed in favor of tests, and can replace them to some extent. But tests can't replace those other factors.
I'm not advocating for zero tests. I am advocating for tests that have the ability to (a) surprise me; and/or (b) teach me something about the problem and/or my solution that I didn't already know. Many tests I've seen are not like this.
Maybe that's a consequence of the kind of code I mostly write? A lot of the business requirements placed on a piece of software are in the form of death by a thousand paper cuts: each requirement is trivial enough to not be worth testing, but it takes a big pile of code to implement all of them and the end-to-end data transformation pipeline is hard to get into a single person's head. In those cases, unit tests are unlikely to find bugs because the requirements are trivial; you really need bigger-scale tests for that. For well-defined tricky-to-implement algorithms the logic is flipped: there you can really get value out of testing just the one function.
I could ramble all day about testing. I'd love to see a test suite which does all the wonderful things I hear about; maybe I could learn something from it.
I have seen tests for bad code also, the average project en with I have worked is a project with bad architecture un completed unit testing because of the lack of time and pressure to make changes, huge refactorings in short time. the work is full of junior programmers that are already learning how to write code, the seniors that write good tests and good architecture are a minority , that is why I think most of the cases unit testing does not add value just quite your precious time.
I think writing tests makes really only sense for a known interface. Because, if you write the test after the code, you are encouraged to the programming equivalent of p-hacking, of testing against the quirks and bugs of the implementation instead of what it should be; but to know what it should be, it needs to be a known interface.
For example, I think it makes sense to have a common test suite for implementations of data structures like stacks or lists, or for file system drivers with operations like open(), read() and write(), since those are known interfaces with potentially very different implementation (in-memory, disk, network etc.). But I don't know if such a test suite would be called a unit test. In a sense, testing push() and pop() on a stack, and write() and read() on a file, and see if the thing written to comes back unchanged, is a kind of unit test, but idk if it would be called that. (I litterally mean code like
push(5); assert(pop() == 5);
I think this could qualify as a unit test, I don't think it's any more complicated than that. The only thing more complicated in different examples would be the examples themselves (like with tree balancing and checking results etc.)) I do think that kind of unit test makes sense.
I happen to work in an accounting software company with lots of banking integrations and even we almost never write unit tests. E2E or endpoint testing is enough to make sure everything works, the rest just slows you down as a dev and as a product company.
How fast and often do the e2e tests run and how reliable are they?
If they function at speed then closer to the exact system behaviour is better. However if e2e are flaky or aren't run often a lot of time passes before you find a regression
summary: do things that you need, don't just do just because...
ding! ding! ding!
I agree only when it comes to UI. I really can't imagine writing security rules without tests tbh.
Unit tests aren't just about catching bugs.. they can do so much more:
1. Prove out the usability of your code, so that it's easy to work with in the future and by other people
2. Structure code so dependencies are minimised/simplified, allowing easy refactoring/extension later
3. Provide living, and correct, examples of how to use the code for documentation
4. Improves/clarifies communication within a team about the technical requirements for a feature
5. Encourages more aggressive improvement of existing codebases by reducing fear of changing behaviour
I agree that unit tests don't normally catch bug regression but they do speed up development in the case where I would normally have to load a page or run it in the console with some sample values testing the edge cases. Otherwise the alternative is to do a bunch of user interaction manually to verify something is working correctly. Much nicer to just save and have the tests confirm it is working within a second or 2. Something about the process of seeing tests pass as you build your function creates small wins and dopamine boosts that make coding more fun as well.
Unit tests also help you refactor. And property testing can sometimes help you make sure you didn't miss an edge case.
Also in dynamically typed languages running relevant unit tests on every file save is a nice way to make sure you didn't break something, especially if you have runtime type validation (almost like static type linting)
Watched to the end and I agree that typescript helps make this strategy viable.
Also - obviously this is different when you develop a public API of any kind (library, public REST api, even a backend that has multiple client applications)
they prevent refactor
@@redhotbits they promote refactoring actually. Ever find your test becoming ugly? That usually means your prod code needs refactoring.
In the best case they catch regressions after future changes.
In the worst case, they provide a false sense if security, allowing other devs to make mad hack Jobs on the code, without the need for deep understanding ... It passes 10,000 unit tests, so it must be good?
Every single system I've seen (in the commercial / web world) is thoroughly riddled with bugs, and all of them already have more unit tests than actual code. So, it follows that the tests can't even be trusted.
@@FranzAllanSee no, thats not how it works. I don't need tests to know when to refactor. If you need tests in order to know when to refactor then you are doing something wrong
Well, unit tests are must in oss. If you don't have it, you will get hurt. It's fine to not write them if you are prototyping solo on some basic non-complex ideas.
But if you build open-source libraries or tools that many people do PR's too you will get burned without tests quite easily. Also, if you build things like DB's, firewalls, parsers or compilers you also must do unit tests. The code in projects like these is not something that can easily fit in your brain when you do modifications.
I can see the issue with excessive unit tests. I think unit tests work well at a certain scale or can be used to limit future problems when there's tension in a system. You may create a test to make sure a distant seldom seen but important part of the system that shares a dependency with your current area of concern isn't broken in some subtle and silent way by a fix or new feature. It's more of an issue when writing high performance high consequence libraries that play around in the dark recesses of the machine. Imagine someone tinkering with the implementation of multi threaded lock free data structures that have the same API and can be compiled for different hardware... there's a security stability data race nightmare scenario you don't ever want to push into production. Another minefield is numerical simulations that use approximate methods but have to converge within known error tollerances and minor changes in basic types, order of operations, code optimizations, or memory management/ data layout can have huge downstream effects on convergence, stability, precision, accuracy or even feasibility... I've seen this happen more than once when engineers tried naively optimizing certain algorithms and suddenly the solution to a differential equation changes by an order of magnitude, without anyone noticing or understanding what happened after it was pointed out... begin 2-3 wks of code archeology to figure out what commit broke that particular system. Everyone was convinced it was a problem with the data or the introduction of multithreading, or some subtle memory corruption... one time it was actually a result of which ALU various parts of the computation were preferentially being sent down... at the time different ALUs in the intel core had different bit widths so you got different truncation errors depending on the alu the processor sent your computation to... the worst one happened where no one suspected: the optimization of a random numerical library that bottlenecked another guy's calculation a year later... Some work requires experience and expertise but very experienced engineers oftentimes forget this fact. The irony of the situation was the original author of the code warned the team not to tamper with the code if you didn't understand EXACTLY how it worked. Solving big equations quickly and at scale is a black art.
Yeah... I don't do 'unit' tests as well. But I always try to do this sort of semi-integration testing. Basically testing a flow, rather than each specific function. But tests are super important. They help you catch bugs before they hit prod when they are done right. Yes, some bugs will still get to prod but then I add another test when I fix it. And also they help you not break stuff you already have. And things break not just because they fragile. Maybe I might have done something completely wrong. The tests help me catch that. Also the value of tests when you want to refactor or improve your code is invaluable. There might be this flow I wrote 5 months ago. And since I know a couple of things better now I might go ahead and improve that flow. If I have tests in place, I can confidently refactor knowing that my tests will tell me if I break any thing. That feedback is super important.
Seems like most of the bugs I catch are from integration/e2e tests.
Unit tests always fail when I change the implementation of a function, not when I break the code
It seems one of the main reasons Theo objects to testing is because of the fragile test problem, where the developer too tightly couples test code with production code, such that when an implementation detail changes but the behavior does not, the tests turn red.
At 6:28 he describes unit tests being like cement and not allowing you to change the code. This is only true if you test implementation and not behavior. Testing only behavior through a public interface/abstraction allows the developer to freely refactor/change implementation details as he/she sees fit, so long as the behavior remains the same.
Also, when you write tests after the fact, usually you end up creating messy production code that is hard to test, which then makes unit testing a horrible and painstaking process.
I think Theo has not sufficiently practiced disciplines like TDD, but instead gave up once he encountered common novice problems such as the fragile test problem.
I think that tests have net positive value when you are in a large team with members of different expertise working together on a repo getting dozens of commits a day. It saves you from manual testing and prevents a regressive commit from getting pushed. My philosophy for coming up with a set of good test cases is first thinking if the code needs to be validated and if so, what validations I would have performed if I were to test manually. I then enlist the inputs and remove any redundancies to get maximum coverage (input-conditions wise) with minimum tests.
And, I get that we need to work on the code if it needs to many tests to stay stable, but we don't live in perfect world. Refactoring barely is an option for large codebases. One man cannot fix the mess and tests become one of the ways to safeguard your piece of code from breaking.
An engineer I used to work with, who I consider to be an actual genius (he’s at Google now), had a conversation with me about this subject, where he claimed that not only is unit testing completely useless, but that unit testing gets people to focus on testing the wrong things, and to miss actual problems. Instead, in his opinion, the only thing that actually works is to write a core set of integration tests that run in a real QA environment, because the things that will actually fail first are things that are incredibly difficult or impossible to test in isolation in unit testing. I think unit tests are only really useful for keeping libraries’ APIs consistent when incorporating changes from a large org, or from the public, ie. in a free-software or open source context. Everything else is window dressing for your manager IMO.
I can't tell you how much time tests have saved me and my coworkers. Unit Tests might not be that useful but functional tests in my opinion are essential.
- Proof of Function
- Easier to Refactor Code
- Reduction of Errors
- Improved On boarding (It would be a nightmare to have new developers work on something with no test cases).
- Peace of Mind
These are good points. However, I do think there should be a bit of a balancing act. In most places I've worked, we are trying to optimize the cost of software development.
the last point itself is a good argument
Unit/System/Iteration Tests are a risk management strategy.
As with any mitigation, you need to consider the cost of the mitigation vs the cost of the thing you're trying to mitigate.
I agree with the take that engineers often misjudge either the cost of the test or the cost of the thing being mitigated against.
My other take on testing is higher level is better. If you have a microservice or lambda function that is supposed to output Y when X is input, that's probably a useful and simple thing to check that's less likely to harmlessly break when an internal change is made.
If your unit tests look like guard rails, if they prevent your changes and if they make you lose time, then you are making them wrong. Unit tests make you blazing faster, help you welcome changes, and are the contrary of guard rails : they foster safe refactoring without even touching the unit test safe net.
Only thing you say I agree upon is that asking for coverage stats is a huge mistake. Coverage does not guarantee anything.
Some tip: you have to test behaviors (black box testing) and not implementation detail. If your tests are using mocks extensively, chances are that you have coupled them to implementation
Second tip: most of your tests shouldn’t be end to end test or tests based on the UI (such as if you click here, this sentence should appear) but should be unit tests on your framework-agnostic use cases (pure typescript). Otherwise you couldn’t change the UI or the framework easily without a full rewrite because of a couple safety e2e net
Unit tests are for public APIs. Whether its a REST API to be consumed by a different team, or an Npm/Cargo/Composer/Pip/etc... package to be consumed by the wider world, your documented features need to be tested so that you don't inadvertently break other people's code when you push a minor update. At this level there really isn't much of a difference between an integration test and a unit test other than the fact that unit tests will mock I/O and will therefore run faster.
Keep in mind you don't need to unit test every internal component. If you have some function that you use all over the place by all means write a few unit tests. It's easy nowadays just ask GPT to do it for you. But otherwise you only need to test the public surface area, not every class or function.
For complex software systems with background processes like queues or cronjobs you should write end to end tests. For the frontend you should use visual regression testing.
When you run into a novel bug that you never, in a million years, would have thought to test; it can point you in the direction of tests that you can write to detect not just that particular bug but for a whole class of bugs that you didn't know existed.
I had to quit my last job when the CI pipeline alone finally exceeded 1hr. So that's a minimum of 1 hour between changing 1 line of code, and getting a green build so that you could be allowed to ask for code review.
After code review (which can take forever), it's another 1 hour wait for an integration build for release, and at least another hour until the release was deployed.
So what do all the engineers at this company do while they are waiting around for CI ? They sit in zoom meetings all day doing "agile rituals" ... and none of them think that is ironic in the slightest.
@@md5ize you assume too much. The CI pipeline here does a build world, and a gazillion end to end behaviour tests that bounce data off flakey mocks that don’t always work.
That pipeline includes JS, php, Java, and god knows what else.
@@md5ize I do agree - for sure, even the thinnest layer of common sense is usually enough to half the time taken in extreme cases.
The problem comes though when the organisation is large and set in its ways, and lacks the political will to improve.
At which point, finding a better place to work is a good option. From that point of view, I went from 1hr CI pipelines to under 10mins CI, and got a hefty $increase in salary whilst I was at it :)
In our Django stack the line between unit tests and integration tests is not that clear, where I would put almost all of our tests into the integration test category. I have never had such a nice integrated testing experience as I had in Django.
Our core application has a big REST API running behind it with different storage backends. There have been at least 3 refactors in central parts of our app that I believe wouldnt have happened without unit tests.
But as you say in the video, this might be related to the domain or your application. Our application lives somewhere at the data layer and bugs there would be catastrophic, so we have a huge test base which has saved us in a lot of occasions.
Also don't underestimate the power of unit tests in an environment where you need to do many things at once. A properly set up CI pipeline can "replace a compiler" if you have proper tests in place. I often find myself writing small changes/optimizations that we don't need tests for (I agree with you that doing things like tests religiously is not a good idea :D) and then pushing them so our CI can check I didn't break the core parts of our app. In the meantime I can do something else.
In the end I want to say this: Unit tests help you while refactoring, not while writing the initial code :)
Unit tests and granular TDD practices at a higher level of abstraction behave like implementation tests. This is only true for larger services or larger individual applications. On smaller, more agile apps, the cost of deletion of an internal module and its test, or even the deletion of the entire application is much cheaper.
The application size of most distributed systems is probably dancing little bit over the edge towards not doing unit tests. So generally, I agree with you.
As a more precise example, if you start your application with a single class or module which does everything that app should meant to do, you start with the test. Which is, by this point, technically a unit test. After a couple of refractor cycles that module will turn onto a network of 20 modules. Breaking up the test to cater to all those 20 modules individually is where the mistakes start to happen. Deeply coupled modules get a "guardrail" between them and cementing the interactions of those modules. That is an incredibly potent source of what-the-fucks.
Testing along the domain lines is much more approachable if, only if there are no dependencies to other domains. But you still need to delete the tests if your problem domain partitioning changes.
I like your path metaphor, but how its being applied has some issues.
1) If we're talking paths and guard rails, you only need to put guard rails where there are drop-offs or other dangers. If you're using safety nets instead, they still have to protect the same drop-offs - and might not work for some dangers. The only real difference is that safety nets catch you after you fall and then you have to figure out how to get back up, whereas guard rails keep you from falling in the first place and you can instead focus on moving down the path.
2) Unit tests are great where they themselves represent requirements, or are written with requirements in mind. Much of the software you're talking about never has requirements in the first place, so unit tests don't apply the same way at all. But they can still apply and your path analogy works: you're building a path and after you get something working you go look for the drop-offs and other dangers to see where unit tests are needed for regression purposes. This process would be improved greatly if your language had an Automated Reasoning tool to help identify problem areas for which you might want to outright fix the code, or instead write some unit tests AND stubs/mocks that limit the use of the unit under test. But this workflow is all about regression testing and providing reminders for future work so that the cost of maintaining your software doesn't outweigh the cost of starting over.
I agree for user-facing apps. One place where I'm more likely to write a lot of tests is when working on a library. I don't want to release an update that causes *your* team to respond to a bug report, track it down, downgrade, and then submit the bug report to me. But even then, it's not unit tests I'm writing, it's somewhere in between unit tests and integration tests. I want to make sure the API exposed to you does what you expect, but I'm probably not granularly testing the internal implementation.
Unit tests are great. But almost everybody unit tests badly and misunderstands them, including Theo.
We only unit test top-most public API interfaces (e.g. controllers or services) and only mock the lowest layer backend (e.g. Repositories). I don't usually test the in-between layers of code, as that code should get exercised during API testing. Testing every single function across all layers, with mocks for every single dependency leads to rigid code that is hard to refactor or change.
Our unit tests are very similar to functional or UAT tests and somewhat similar to integration tests, and therefore we don't need functional or integration testing. (However, we have a few smoke tests to make sure everything is wired together correctly.) Our tests are blazingly fast, and comprehensive.
i made a case study in unit testing / B2B test for many use case, and i concluded that without testing, deployment is awful.
Here is why : the reason for testing is to save time. however most people think that testing takes huge time so they dont test at all - and they'll end up spending time debugging and manual testing that almost takes forever at the end which goes back to why testing exist is to help us save time so we can spend time doing other things.
Could you make a video about your pipeline and how you go about designing and implementing the safety nets please?
The shit thing about unit testing is that the 'unit' is mostly defined as microscopic as possible which invariably leads to implementation details being tested. This implementation detail testing is what pours concrete on your current implementation, and makes tests a liability instead of an investment. Because you cannot rely on your tests to tell you whether the expected application behavior is intact, but it instead fails when the behavior is correct just because its implementation details have changed. Makes a world of difference. In a sane world 'unit' would always be a unit of expected application behavior / use case, and tests would be an investment which actually encourages code changes and improvements, because it would provide reasonable assurance that the refactor works great.
I am one of those folks who does financial services. Here's my take on unit tests:
Unit tests can be great to help you refactor with confidence. My reasoning for this is that if your integration tests are failing, a well-written unit test suite will likely surface the issue more precisely. Integration tests will tell you "X thing wasn't what you expected", but unit tests tell you "X thing wasn't want you expected and this is exactly why". There's often less debugging involved when you have a well-written suite of unit tests.
That said, most of the tests I write are integration tests because testing things in isolation will only get you so far. Mocking things to oblivion isn't effective either.
100% agree with not cementing things in stone. If there's a smell in your code, burying it in tests will not fix it.
I feel the poster, having seemingly not written unit tests does not actually understand the benefits.
The cement for the unit tests is only the first phase. Once you documented all of the test outcome, then rebuild the thing. then you can verify the results are the same after you rebuilt it.
Unit tests are great for asserting that some logic, a calculation for example is working as intended. I use unit tests for all such logic (and validators).
I don't see much of a point in using them for anything else. I think unit tests are good in general, because then you'll know when everything is working and such, but in many cases I find the unit tests take far too long to write.
In the last case you mentioned, unit tests make sense if you wish to rewrite the code. Then you can let the old code stay and assert equal outcome in the new code that will hopefully be less fragile.
After that is done and it has been properly tested in production, I'd say it's time to delete the old garbage and the unit test however.
That is one of the ways I've used unit tests when working with very sensitive things; get the result you expect, once test passes, your code is working. Do some final refactoring to clean up a bit, then you're done.
One thing I do keep as a principle however; if your code can't be unit tested, you should rewrite it. Not for the sake of actually unit testing, but it shows that each segment has too many responsibilities.
I confess that my experience with unit tests is limited. Early on I wrote unit tests that were actually integration tests because they connected to the database. This actually seems like a more useful type of test to me, but they take a long time to run so they are a pain to use for that reason. I am biased against unit tests because most of the apps I work on are essentially document management type solutions that are web-based applications. You cannot write unit tests for javascript, which is used to a large degree, without using javascript unit testing libraries. You can't use the standard C# unit testing libraries for an HTML/Javascript UI. Fill out a form, validate the form, insert/update/select from the database. If you are building a space shuttle and you have scientific calculations that must be exact, I can see the benefit of unit tests. Otherwise, I find them highly annoying and not useful for the type projects I have worked on. The idea of mocking out the database and testing is so bizarre and time-wasting to me. Maybe someday I will work on a project where I see the benefit. So far, I have not. Lots of overhead and time spent on something that seems ridiculous to me.
Agreed. Most of errors in software comes from business, from analytics who don't understand the world, the domain correctly. Probably there's some use if high-level developer/analyst wrote tests for lower-tier developers to implement code for. This will ensure the business domain is understood better than average developer could. Also, most of software we buid are just fancy CRUD interface for databases. Read/Write to database is a side-effect and there's little sense testing side-effects.
unit tests do not slow you down. they save you time by preventing you from breaking everything and having automated documentation of how your codebase works
Yes and no. I believe the correct conclusion is: it depends (as is usually the case in tech). There is a tension between the cost to do unit testing and the cost of the time they save you.
if you're writing a component, and you don't write unit tests you will miss corner cases. For example, if you're implementing a new data compression algorithm, and at some point you discover that you need a specialized variant of a circular buffer, you're going to want to write down *explicity* the behavior that is expected for that component. You should also write unit tests for every single one of those requirements, that way if someone ever modifies that code, and violates one of the requirements you'll know immediately that: 1. something is broken, 2. what is broken.
Other times that you *will* need unit tests are (in java at least) when you introduce new data types that you intend to use with the various Collection classes. It's very easy to add new fields to a class, and forget to update equals() and hashCode(). Even if you remember to update them, It's easy to forget about the contract between them, and update them in ways that violate the contract between the two methods, this can lead to very strange and difficult to diagnose bugs
Every class you create should have a purpose, and documented, defined behavior that is unit tested for violations. If you're just whipping out new classes without thought or documentation, and without tests then you're going to run into situations a class is used in a way you didn't anticipate. The code may still work correctly by coincidence, but later on as you make changes you may find you've broken hundreds of other components that relied on that coincidence. At that point you have to go refactor all those other components not to rely on the undocumented behavior.
Unit test that exist as TDD not written after the fact tend to document the developers intentions as they built the code, which could be useful as documentation and help future engineers that come on to the project, even when the dev that built the software has left the company. It is awesome for a dev to come in, study the unit tests on the feature they are working on, quickly see how the code in those sections is written and how it runs, so they can be productive fast. It also helps new devs know if they introduced a bug if the change they were working on wasn't supposed to break existing functionality. You said something awesome in your video, if management focuses on unit test coverage, then they are probably forcing their devs to write garbage, if devs are free to write a unit test or two that focus on the things that matter in combination with code reviews/PR's, then the tests tend to be better and more useful.
I feel like i agree with you in part, but i also feel like there are some pieces of code that can be most conveniently tested with unit tests, particularly well contained pieces of logic with few dependencies. I can often test more cases of my algorithm more quickly and end up with a test suite that runs in much less time in the cases than if I'd tried to write integration tests to really try to exercise the algorithm. These tests are both useful when developing the algorithm initially and also as i try to apply optimizations over time.
That being said, i think that all testing requires the engineer to be smart. You need to be very careful not to overspecify the test or to try to test things that the type of test you are writing isn't good for.
100% agree that having a unit test coverage threshold is a terrible idea, though.
Lovely hot take :) The reason TDD, Scrum etc are so extreme about applying them fully is of course the same reason there are no successful vague "do what's best" diet plans. People need to be sold on some magic false promises to get the placebo effect to actually make changes in their lives, and to enforce healthy routines for their team communication. In reality having or not having tests is a trade-off and sometime it's just not worth it.
"Covering it in cement" is exactly what I've been saying about unit tests also. It's especially true for poorly written (even generated) tests that's literally molded after the function output at the time of writing, decided that was the expected behavior now and forever. Real TDD of course tells you to write the tests first. That is also unrealistic, because more often than not you will realize in the implementation that you need to change the design you originally envisioned due to REALITY.
People also say that writing tests is documentation. But you shouldn't treat tests like documentation. Let's say that your tests says a matching blocks function taking the input "Web Browser" and "webr" giving the output "[We]b [Br]owser". Someone rewrites the algorithm to be faster, but now it outputs "[Web] B[r]owser" or "[Web] Browse[r]". You don't actually have the documented intent for the first implementation. You just have a test failure and don't know if it's because the original test made an unintentional assumption, or if you broke an important rule.
"Covering in cement" is an example of a degenerate practice that is typically the result from the misunderstanding that microtesting is about "correctness". Microtests are about near instant feedback. So those tests should cover observable behaviour (protocol) that will not change regardless of implementation. Once those tests are in place you go ahead and replace the legacy implementation with the improved one while the microtests are your safety net for sticking to the required protocol. This is the core of
"For each desired change, make the change easy (warning: this may be hard), then make the easy change".
Your "[We]b [Br]owser" example simply highlights an incorrectly implemented test. If in fact "[Web] Browse[r]" is acceptable, the test should be checking for the sequence of characters occurring between the square brackets-not "[We]b [Br]owser" specifically. This goes to the point that James O. Coplien made in "Why most unit testing is a waste":
"Design a test with more care than you design the code"
i.e. writing good microtests is harder than writing good code. Simply learning how to use a unit testing framework (or worse, a mocking library) does not teach you how to write good microtests.
Theo is using his continuous deployment pipeline and his users as his feedback loop. But that also means that encountering and reporting bugs becomes part of the user's product experience regardless of how quickly Theo may respond with fixes. That's fine if he has a customer base that is tolerant of that practice but it's much more likely that clients will simply get frustrated with the defects, not report them, and just leave.
UI -> Integration and E2E
Structural and Creational Code (helpers, builders, factories, adapters and services) -> Unit Tests
This will help with Maintainability and Readability.
If there are no unit tests, refactoring large pieces of structural functionality will become difficult and might cause mountains of Tech Debt, seen this happen especially when Migrating large pieces of work.
I think it also depends on the level of safety or security the systems needs to be. I am sure an astronaut would want a space application that handles a launch of spaceship to be tested to the extreme. Similarly for banking or transactional systems, the banks would want the tests to be as stringent as possible in terms of security to prevent their accounts from being hacked. We probably also need not write full coverage on the entire system, just those that matters
I completely agree with you Theo. I've been programming for 20+ years and haven't written a single test at an employer. I experimented in my spare time for curiosity sake. Almost all companies I interview for bring up unit tests and then when I start there they don't cater for unit tests in their sprints, and don't even have a culture of writing tests.
The ONLY tests that I've actually written and seen true value for is E2E using tools like Cypress. That has definitely help catch issues/bugs. Cool thing is an E2E tests covers both FE and BE and not JUST your code base.
I also like to write Postman tests to check for API changes, cuz you know a LOT of BE devs don't bother telling the FE devs when they change/break something...
TDD is for long term large scale enterprise application devlopement, if my app is some components, I don't test everything, it's overwhelming and time consuming for a simple app. If you're building an app in a team for thousands of users, you'll have time for implementing every component and test them, to be able to move to another thing.
Could you elaborate more on how to build the safety net?
Plot twist: Unit test
I'm primarily a C# dev and have long held the purpose of unit tests is to enforce reasonable coding standards on junior devs. If you force devs to unit test you're kind of forcing them to use dependency injection ... otherwise "testing" will be hard. You're also kind of forcing them to break down their code a bit more because 1000 line methods are hard to test. In a dynamically typed language sure unit tests have a place and typescript addresses those well.
Unless you're working on some kind of calculator where you can capture requirements very discretely ... or if you don't know exactly how to approach a problem writing some unit tests and doing some TDD is really helpful. These situations are extremely rare for experienced devs.
Totally agreed. I even skipped a job opportunity because they forced me to to unity tests. At that time it felt stupid and still does. It just complicates the whole workflow and eliminates many essential possible solutions. Also at that time it made me think they have some deep bugs in their solutions that they are trying to avoid/hide. I am also more on the electrical side of education, so I tend to keep the code as efficient as possible, because lots of "ideas" are not actually good for the electronic circuits on the physical computer or a server.
I believe that it depends on what you're trying to achieve. One may argue that TDD slows you down because you need to also write a sustainable test suite, but one could also argue that code without an automatic way to test itself would not only slow you down, but it will stop you forever. i.e. there can't be any safety net for you to go further, and competition will win the race.
I also believe the problem with TDD is that needs time and practice, and many people can't or won't afford that.
I mostly agree, but unit testing can be sometimes useful when you want to see if a fuctions works properly whithout running the whole backend,frontend database enchilada.
It is funny because Theo is using removing those safety nets as an example why you should not use unit tests
I find that unit tests are useful when your goal is to build and optimize a complex algorithm (which has nothing to do with business logic). Typically something you could put in a library, like a graph algorithm or a sorting algorithm.
Once the optimization is done, you might as well delete the unit tests - but they can be useful to keep around in case you need to optimize more in the future.
In the case of ensuring correctness while optimizing - and also testing performance, it often makes sense to keep a slow/naive/simple version of the function around, and instead generate random test data (with some seed to make it reproducible) to check that your optimized version always produces the same output, and actually speeds up as you modify it.
I've always felt this way about unit tests. The only time I really found them useful is if I wanted to write something in an isolated, data-in, data-out kind of way more rapidly. Not for the purpose of getting 100% code coverage. In other words, if it actually boosts your productivity, then unit test.
My department ended up in a sort of compromise where we basically only end up writing Integration/end-to-end tests.
Do mocked request, backend does stuff, check that the data is in the expected state.
Good test approach shouldn't slow you down. But it requieres practice and compromise! If you do it well, it gives you real confidence and you can change and adapt the whole thing in a whistle
Agree that unit tests are not the best way to catch bugs in systems. I find assertions/invariants much more effective (specially when dealing with data coming from outside the code - eg. database/http). That being said, if a unit test feels like cement to you, might be a smell that you are testing implementation details? But it all depends on what you're working on. Unit tests are really useful for problems where all the logic is contained within the codebase...rarely the case for modern web based projects. Almost always the case for Frameworks/Libraries/utilities.
I think of unit test as a documentation. it should tell what this part of code allow and does not allow. but it should be used only for complex stuff. We devs are lazy so there would always be bugs. We should just be ready to fix stuff as fast as possible.
I found myself testing less and less over time.
But i still test the hell out of regular expressions. I love regular expressions, but they're impossible to read and worse to debug. My testing is part experimentation log, part example, and almost 0% regression testing.
I feel it's about the cost of failure/bugs in our code - how does it impact our business ?
At one end of a spectrum, we may have mission-critical apps that have an extremely low tolerance for failure. On the other end we could have a simple application, where the business could be more forgiving with the failure.
Even within Apps themselves, there could be parts that are super critical and some that are less critical.
The approach will depend on the criticality and the complexity of the application and also how long we expect the application to last (shelf-life)
i've said this at work so many times, that if a test breaks it's because the change i made to the code intentionally broke it, so my job then doesn't become fixing the unit test but it become creating it a new to fit the new scenario. Never ever have i seen a test fail where it wasn't supposed to so why on earth do we waste time on them.
I'm so glad I found your channel a couple of weeks ago.
Cheers!
I don't know if this is same thing but the bootcamp I am currently on is all about tests, red green refactor and it's so hard
Another value of unit tests is giving the author feedback on the design of the unit. If it's hard to unit test, it's an indicator that perhaps the unit is too tightly coupled with other things.
I like this take. Can you offer some examples of "Safety Nets" for clarity?
TDD, ATDD, and TCR are great approaches to add a safety net. If you work in baby steps, you end up going faster. It’s the same principle as measuring twice and cutting once. I’ve always found not writing tests ir writing them after to be the same as not measuring or measuring after cutting.
XP and Clean Code principles will make your code safe, easy to maintain and extend.
Good unit tests doesn't make code rigid they make sure that the acceptance criteria are, and you write them to prohibit you from changing them unknowingly. Unit tests can be changed, and actually have to be changed when the acceptance criteria change.
As some have pointed out it really comes down to what you are testing and how.
Do I need a test to assert function x has been called y times and with arguments a,b,c? No. Do I need to know that my function returns the correct answer for my input? Hell yeah.
I think a lot of the aversion when it comes to unit tests is coming from not doing them correctly. Test public and/or critical parts, test behaviour not how something is done. In my opinion if a test is holding me back, it was done improperly.
This brings me to another point. Tests are an incredible tool for identifying code smell. Each time I've had to take a part of a codebase that didn't have unit tests or wasn't setup properly it was so helpful to write proper tests in order to make things more flexible.
One bonus is if you have a really good test suite for your service, then you won't need a lot of documentation. Tests can be a better tool when it comes to explaining how a codebase works and also why decisions were made.
It takes time to get a team to a point where this works properly but I think it has a huge return on investment.
Tests pay for themselves over time. They pay for themselves because they allow to refactor and change code in non-trivial ways without the fear of breaking something else in weird and subtle ways.
Also many applications are structured like a pyramid - some code lies at the foundation and enables layers and layers of complexity built on top of it to work correctly. And some code, mostly UI code - lies at the very top and interfaces the user directly, there's nothing further built on top of the UI code. Tests are exponentially (and I really mean exponentially) more important, further down the pyramid. So skipping them for the UI might not be such a big deal. But having a spotty coverage at the foundation will make any non-trivial change extremely expensive and risky. Not to mention if you have multiple undertested layers in your application, then guaranteed some of your features are completely broken, you just don't know it.
It's also about how non-trivial a mistake is. The most valuable tests can uncover subtle issues that your users might not necessarily uncover. What if an error in your application causes incorrect stats to show up in the analytics panel? What if an error in pagination causes the list to be trimmed by one item? Are your users really going to notice and point that out right away? How many months of erroneous data is going to be aggregated, used and relied upon before someone uncovers the problem. Not every error is an exception or a database query failing. What if a 3rd party integration starts relying on that erroneous behavior?
Also understand that Facebook has multiple internal teams fixing other people's code as a continuous process. So yeah, maybe that one engineer did not write tests. Very possible someone else fixed their code or wrote those missing tests for them. For instance code quality teams routinely remove millions of lines of dead code that was written by others but due to programmer error never (sic!) run. Not even once. Is writing, reviewing, maintaining dead code really an efficient use of company's resources? It makes one think.
The right tests will increase your engineers velocity, especially as the application grows. Teams that don't have a good culture of testing often slow down over time and have to over-hire which is a hidden cost.
Strongly disagree.
Unit tests saves a lot of time in bigger projects. They are designed to quickly check if changes havent broke stuff. Unit tests cover core paths - what must be working after each commit. Also - safety net for errors in prod.
Even if you are solo dev in the project, how sure you are that your changes won’t cause global issues in the project when project scales?
This is like statement that typescript is useless. When you see the bigger picture - you understand why ts & unit tests are very important.
It takes time - sure it does. But it saves tons of time in the long run.
As usual Theo makes some great points, mixed with some misleading ones. There are a few more scenarios I would highly recommend good test coverage!
Would love to see a more in depth discussion on the scenarios where different types of tests are useful... as well as Theo's take on tools like React Testing Library, which I've had mixed experiences with.
Hey I'm all for unit testing, but I can't say I'm a fan of React Testing Library. The whole way unit tests are done on the frontend, they are just so cumbersome and difficult. I can understand Theo's point of view because he is mainly a frontend coder, and I've never to this day had a good unit testing experience on the frontend.
This is Theo's MO--using inflammatory, over-generalized statements to stimulate discussion. The intent is to challenge the assumption that unit tests are always necessary/beneficial, not to actually do away with unit tests.
@@DanKaschel Yup. I love Theo's style! I just hope junior developers understand what you just said above 🤣 I loved his debate about unit tests with the Netflix guy. Fantastic content about the pros and cons.
I definitely agree 👍 unit tests really just make things harder in many ways.
Unit tests are valuable safe net when new engineers join your team and start to mess around million assumptions you made in your head. If you never spend the time making unit test to "document" these assumptions, you will have to spend 10x the time debug when other teammates break your code. I find most people who argue that unit tests slow you down is from companies that have
I would like an overview of what is possible as a safety net because I feel like this is not a common topic. I totally agree that the need to have easy and fast code to production is very important for the developer experience, specially for junior
1 million percent true. We work in an industry where testing has become a dogmatically enforced and viewed as an unconditional virtue instead of how we should view it: as specific tool to solve a specific problem, apply it where it makes sense.
I really like the idea of using feature flags so that if things go wrong in Prod it can be fixed fast.
For me the reason writing test cases is pointless is because
1. Unreasonable deadlines set by management.
2. Most of the time the test environment doesn't adequately represent the Prod environment. This means that when you find issues with you code in Production. Now you have to go back and re write the test cases as well. But you need to fix the prod issue as fast possible so the test cases actually becomes a hindrance.
3. Ever changing requirements by management. TDD depends on this idea that you have perfect information. But if requirements keep changing you have to re write test cases which defeats the whole point of testing.
I am for writing test cases but in the modern world things are never that simple.
I'd be curious what pipe you are using that you refer to - what sort of monitoring, feedback, and ops pipelines are you utilizing?
Most senior devs understand that Code Coverage requirement is nonsense. It usually comes from managers who saw some presentation about tests and decide "more test better, 100% test best". Adding test coverage to existing code is almost always a waste of time.
This doesn't mean tests are useless or harmful. TDD development, if done by people who understand it, means writing tests to cover the requirements first, and then implementing the solution to make sure the code does what it's supposed to. It usually leads to cleaner code with some intelligent mocking at unit boundaries.
I don't have much experience with UI testing, from what I saw they are usually slow and not much safer than the code they are testing, maybe that's the problem to solve for framework creators. But in the backend code, good tests are saviors both short and long term.