Testing Strategy for DevOps: What to Test and When
Вставка
- Опубліковано 4 чер 2024
- Your testing strategy is central to your ability to deliver better software faster. Excellent teams have a laser focus on testing, and this focus extends way beyond what we do before we release our changes into production.
Testing to support DevOps and Continuous Delivery is a core idea, and testing in Production is part of that game too. So what should we test and when should we test it? What kinds of testing techniques should we adopt?
In this episode Dave Farley will describe his take on what a holistic test-strategy looks like, and gives us examples from the real world to describe it.
-------------------------------------------------------------------------------------
LINKS:
Cindy Sridharan’s blog on ‘Testing in production the safe way” ➡️ / testing-in-production-...
The Netflix Simian Army: ➡️netflixtechblog.com/the-netfl...
"2/3 of ideas produce zero or negative value" from "Online Controlled Experiments at Large Scale", a Study at Microsoft ➡️ bit.ly/ExPScale
-------------------------------------------------------------------------------------
🎓 CD TRAINING COURSES 🎓
If you want to learn Continuous Delivery and DevOps skills, check out Dave Farley's courses ➡️ bit.ly/DFTraining
Get a FREE copy of Dave's "WHAT TO TEST & WHEN?" Guide.
Sign up to the CD Mail List to keep up to date with the latest discussions, other free "How To..." guides, events and online courses.
➡️ www.subscribepage.com/what-to...
📚 BOOKS:
📖 Dave’s NEW BOOK "Modern Software Engineering" is now available on
Kindle ➡️ amzn.to/3DwdwT3
(Paperback version available soon)
In this book, Dave brings together his ideas and proven techniques to describe a durable, coherent and foundational approach to effective software development, for programmers, managers and technical leads, at all levels of experience.
📖 "Continuous Delivery Pipelines" by Dave Farley
paperback ➡️ amzn.to/3gIULlA
ebook version ➡️ leanpub.com/cd-pipelines
📖 The original, award-winning "Continuous Delivery" book by Dave Farley and Jez Humble
➡️ amzn.to/2WxRYmx
-------------------------------------------------------------------------------------
CHANNEL SPONSORS:
Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ www.equalexperts.com/
Harness helps engineers and DevOps teams simplify and scale CI/CD. Sign up for your free account at ➡️ harness.io
Octopus are the makers of Octopus Deploy the single place for your team to manage releases, automate deployments, and automate the runbooks that keep your software operating. ➡️ octopus.com/
SpecFlow Behavior Driven Development for .NET SpecFlow helps teams bind automation to feature files and share the resulting examples as Living Documentation across the team and stakeholders. ➡️ go.specflow.org/dave_farley - Наука та технологія
the main factor for me to join ThoughtWorks was knowing I would be able to work with and learn from professionals like Dave, I'm starting next monday as a QA Engineer
Lucky you 👍🥳
QA is one of the most important things you can do. Don't let anyone tell you otherwise. A good QA engineer develops unique and specialised skills that other developers typically don't learn.
Are you a
QA Engineer - manual tester of business applications
or
QA Engineer - test automation specialist?
Thank you. Good luck at ThoughtWorks, I hope that you have some fun.
@@cdarklock thanks man, I actualy help positions of frontend and backend development in the past, which only made me understand (sometimes the hard way) how important is to have a committed tester by your side
You've quickly become a favorite for me. Thank you for your insights
Thanks
Same here 😊
Excellent content as usual, so helpfull, thanks Dave for the amazing job of making it so simple when you say it ;)
An excellent summary of testing strategies!
Outstanding content, as usual.
Thanks 😎
Thanks Dave! Could you point to the source/article of you mentioned that according to state of devops report says that 60% of critical bugs could have been prevented by unit tests?
Wasn't SODR it was this: www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
I am an Android focused engineer, but today anymore, I think if your not doing CD, your working towards it.
Or should be... ,
I can see that now, even in smaller organizations what a huge benefit it would be to have a continually deliverable and releasable product every planned iteration...I am personally loving it.
I'd love to hear your take on delivering Machine Learning projects.
Hello, do you have experience or plan to do any videos in regards to video game development and best practices in the software of the video game industry? A lot of your concepts of course are perfect for backend/server side development of games, but fast iteration and test coverage of client side stuff is very difficult.
How is canary testing implemented in practice? Feature toggles based on for example local time, in your Netflix example?
Great stuff
Great work Dave! I'm a QA engineer myself and always looking for nice vids to improve my knowledge and I can tell you I'm totally in line with your story. Thats why i started following you! Keep up the great work!
I was just wondering. I've seen the sheet you're using somewhere else but i can't seem to find it. Do you mind sharing it as I might want to use it as a reference sheet to help customers out improving their QA process. I'm aware not everything on the sheet will be usable with every customer but it's always nice to have some guidelines.
For my description I don't have a 'sheet' - yet, but I probably will release this as an information sheet to people on my mail list, if you sign up for my ATDD guide, here, we'll let you know when the info sheet in this video is available. www.subscribepage.com/cd-atdd-guide
The one from Cindy Sidrahan that I mention is already linked in the video description.
This is good stuff Dave, this is good stuff!
Thanks.
Could you talk around the testing in prod part and if you keep consistent in your testing and keep it in your logging and db or do you exclude the data points or records they create.
I think it depends on the nature of the testing. Some of the kinds of testing I talk about are consistent, certainly the release-focused testing is largely the same and run on each release. Some of the product-focused testing is too, probably the more technical kinds of tests and monitoring that you run. The very product focused testing like A/B or other kinds of Biz performance monitoring are probably more targeted and variable in time. The best way to organise this is to think of each release as a small experiment. If that's the case, then part of each new feature is "how do I collect the results from this experiment?" so you build-in the case-specific data-collection to the feature.
Great video with many good ideas for testing in production, will definitely be coming back to it!!
Though there's a lot of mentioning of confirming and validating in testing but there's something niggling in the back of my mind that doesn't sit right. Surely the point of testing is to disconfirm and invalidate? Actively look for ways in which it doesn't work or deliver value, not just focusing on how it does. Otherwise you open up yourself to confirmation bias and might be missing huge areas of unknown risk and failure? I think the video hinted at this with some techniques, but only lightly.
Yes, me being lazy with my language. Testing is best treated as a "falsification" not a "validation", 💯!
@@ContinuousDelivery Yes! Well said. That's the tester's mindset I was trying to think of. Thank you 🙂
Thanks, great as always :) 5 minutes for commit tests raised an eyebrow, though - I think depends as always on context but if you're committing say 20 times a day that's 100 minutes of potential flow interruption presuming no re-runs... seems high and 5 minutes definitely feels high to me whenever I've had to deal with it... Faster is better all other aspects equal, it's good to aim low, imo... Maybe a video on build times and various cycle times effects? Anyway, thanks again.
Thanks. Of course "it depends". For some systems "5 minutes" sounds a ridiculously short time, for others it seems way too long. I think that 5 minutes is a reasonable upper bound, but always prefer it to be faster.
I like your videos, thanks for your effort!
Thanks
What is your experience with scheduled releases?
Do you prefer weekly releases or bi-weekly releases in general?
Do you try to limit the PRs that come into each release to control the technical scope-of-change happening each release or do you let the product/project manager decide what can go into every release?
In the place I work in, we would release every week, first 3 days is for merging and testing the PRs ( project manager decide what can go every week ), next day to run stage-deployment and internal users to test, last day for production ( Friday )
So far it is working well for us, sometimes we would move production-release to Monday if the release is critical and need extra care.
Personally, I don't have much experience with release control, so looking to learn more from you regarding this aspect.
I understand the endgoal should be continous delivery ( PR merged automatically goes to stage, press a button and production is updated ), we are working to reach it but still a while away.
Thank you for the great videos! Heard you on the ChangeLog podcast the other day and enjoyed hearing your insights and opinions thoroughly.
I have worked on teams that did scheduled releases at various frequencies. I that that some of the costs that you mention, like “3 days merging and testing PRs” highlight some of the problems. In general my advice is, even if you run a scheduled release approach, the best way to do that is via Continuous Delivery. That is, work so that you software is ALWAYS ready for release.
Shorter release-cycles are better, because they force you to address pain-points. If you sped-up, you would be forced to address the “3 days” and replace it with something better - True Continuous Integration is the best approach, getting to something that works at least daily. CD is the next step after that, getting something that you “could” release at least daily.
Hello, great content. Is it implied that u don't run the acceptance tests on each commit? Should we wait a number of commits or "overnight"? How does that align with having your code releasable all the time?
No I prefer to run them all the time, but not necessarily on every commit. I divide deployment pipelines into 3 phases, commit, acceptance and production.
The commit phase runs on every commit and is optimised to give fast feedback. Its job is to keep development flowing.
Acceptance is about determining releasability, tests are more complex, more "whole system" and so slower.
Prod is about release and monitoring and gathering feedback from production.
Let's imagine commit takes 5 minutes and acceptance 50. So during a single acceptance test run, we may have processed 10 commits. When the acceptance run finishes, the acceptance test logic looks for the newest successful release candidate - generated by a successful commit run, and deploys that and runs all the tests. So Acceptance testing "hops over" commits, which is ok because they are additive, the 10th commit included all the new features of the previous 9, so acceptance testing is testing everything as frequently as possible.
@@ContinuousDelivery smart, why didn't I think of it :).
7:55 okay, you realse something for a day to the green marker, and only release it to the yellow market on the day after. How does that correlate to continuous delivery. If I write a new feature, do I also only release it to green and not to yellow? I presume, if I add 10 more features that day, I should release them only to the green market? Does this mean that my feedback loop to the yellow market is 1 day instead of 5 minutes?
Production is the best environment for destructive testing! :D
Probably not for pacemakers 😳 but it is certainly the most realistic.
Hello Dave, Love your videos :)
Can you tell me how/what you measure the level of confidence that you talk about in the commit cycle?
In the case you say 80%, but what are you using to composite this number? E.g. Code coverage, mutation score and etc
Thanks
Easier than that I think, I mean that if all the tests pass in the commit stage, 80% of the time, all the tests will pass in every other stage of the pipeline too.
@@ContinuousDelivery It really was easier than I imagined and it makes perfect sense. Thanks for the answer :)
I've been doing a little of CI and now I'm learning CD.
Would it be ok to say that the Commit Cycle is the CI section of CD?
Yes, that’s how I think of it.
@@ContinuousDelivery thanks!
❤️
quality
Hi Dave, automated acceptance testing is intriguing to me, you can almost call it an obsession. One thing that has me confused though is how it relates to the test pyramid. I believe often 70% unit/20% integration/10% e2e are mentioned as guidelines for the distribution of types of tests. However, when for each story we create e2e automated acceptance tests, the picture may look quite different. So my questions are: Should automated acceptance tests always be e2e (via UI and/or API)? And what does it do in practice to the shape of the test pyramid? Or did I misunderstand, and are automated acceptance tests usually testing a "logical" service (one bounded context, one vertical slice) full-stack, but mocking external services, and services for other bounded contexts?
Not sure if I understood you correctly but doesn’t the 70/20/10 testing proportion relate to the proportion of tests for EACH FEATURE released not a proration of all tests? This, I thought, means that when delivering a feature 70% of the testing effort should be unit tests, 20% integration and 10% E2E. So, the developers create most of the tests with the final QA being a few E2E tests, (these often being the most brittle). All these tests being automated, of course. Apologies if I got the wrong end of the stick 😳
Hi @@portlyoldman, thanks for sharing your thoughts. It leaves the question though, where in the e2e pyramid do these "automated acceptance tests" fall? From Dave's videos so far, I understood they are e2e (but mocking external systems). And that's what is confusing me. On the one hand there's the test pyramid which I respect, on the other hand there's this channel which I respect (there's a lot of terrific advice here), but these two seem to give slightly opposite advice. If we write automated acceptance tests/executable specifications for each story, and those tests are advised to be e2e, then doesn't that produce too many slow, brittle, and hard to maintain and troubleshoot e2e tests? (And that would be problematic for a CD pipeline)
@@portlyoldman and Dave, Here's another view from martinfowler.com/articles/practical-test-pyramid.html: "Acceptance tests can come in different levels of granularity. Most of the time they will be rather high-level and test your service through the user interface. However, it's good to understand that there's technically no need to write acceptance tests at the highest level of your test pyramid. If your application design and your scenario at hand permits that you write an acceptance test at a lower level, go for it. Having a low-level test is better than having a high-level test. The concept of acceptance tests - proving that your features work correctly for the user - is completely orthogonal to your test pyramid."
Do you agree with that view? Can acceptance tests be written at lower levels of the test pyramid? Or is it then no longer an "executable specification"/"automated acceptance test"?
@@gruttewibe76 - thanks for the link, I shall read that with great interest. Like you I’m obsessed with testing and software proving in general 🤓
I suppose it depends on how far you take unit testing and what you mean by the percentages. For a feature I'd generally expect to create a handful of acceptance criteria and an automated "Acceptance Test" for each. If you take my approach, most of these "executable specifications" will reuse lots of test infrastructure code and will usually add little new code. The test case itself is usually a few lines of code written in your test DSL.
Unit testing is driven, for me, from TDD. So I'd create unit tests to support nearly all of my code. So I'd have quite a lot more code in unit tests than code in acceptance tests, though the testing infrastructure code for acceptance tests will be more complex.
One that basis, in terms of effort then something like 70% unit vs 10% acceptance is probably about right, though a guideline rather than a rule to stick to.
If you count tests, then I think it is harder to generalise. Some features, may already exist by accident, so you will write an acceptance test to validate the feature, but don't need to write any additional code or unit tests. Unusual, but I have seen it happen. Other code may need a simple acceptance test and loads of work, and so loads of unit tests, to accomplish.
I confess that I am not as big a fan of the test pyramid as some other people, in part for these kinds of reasons. I think that it can constrain people's thinking. However, if you see it as a rough guide, then it makes sense. I would expect, as an average, over the life of a project, for there to be more unit tests than acceptance tests, lots more.
The danger, and a trap that I have fallen into on my own teams, is that the acceptance tests are more visible and more understandable, so there is a temptation to write more of them. QA people for example, often say to me "we can't see what the devs do in unit tests, so we will cover everything in acceptance tests". This is wrong on multiple fronts. 1) it isn't the QA's responsibility to own the testing or the gatekeeping 2) its an inefficient way to test 3) it skews the team in the wrong direction, if the QAs test "everything" in acceptance tests it will be slow, flaky and inefficient but it will nevertheless tempt the devs to relax their own testing, and abdicate responsibility to the QAs.
Ultimately I think that unit testing is more valuable as a tool, but that acceptance testing gives us insight and a viewpoint that we would miss without it.
I agree and disagree with BDD style acceptance tests being where they are (acceptance cycle). If we have BDD style hexagonal core tests and contract tests in the commit cycle and then something called integration tests in the acceptance cycle then our confidence is already very high. Only if a project is incredibly important to not be deployed if something has small percentage of chance of going wrong then I would repeat the same BDD tests in the acceptance cycle (this would require new implementation of test DSL). Even then, deploying to the production with canary deployments would help find some problems fast (if we can cause problems to a small group of beta users that is). The rest of the problems would be taken care of by feature flags which would be enabled for a small portion of the users if deployment worked.
I just never found a good reason to run a lot of BDD style tests if everything else I described is done well.
But I guess if we write tests in a right way, can prepare two test DSLs (usually both implementing the same interface but doing things differently underneath) and also can make those acceptance tests run very fast for various adapters for each port (as in big systems there could be more than one), then there is nothing wrong at all in having acceptance tests in the acceptance cycle. Actually, if they can be run on local without problems and run extremely fast we can relax on how many unit tests we create to not have to duplicate too much of the test code which test the same things (but then developers would have to remember to run acceptance tests locally which I doubt is always possible).
I need a bit more experience but so far I am not sure if I like your way or Cindy's way more. Maybe I will make up my mind someday when I get more experience.
Ah, and great video. What you say in it is spot on. I learn a lot from you. Thanks!
Why does it feel like you have completely missed the part where a software testers tests the feature by hand?
Or am I missing something?
"If it works it's Production, if not it's a test"
Wish I had someone like you on my team, rather than 3 guys all hired directly out of school with no experience outside our current placement, all trying to keep a billion dollar company running by reading and watching things that we think might help... but sometimesend up hurting.
You can't have a one size fits all software testing process.. it doesn't exist, if anything you've got this whole release process back to front
Skip the first 5 minutes. Misleading titles,
Funny you keep mentioning Netflix, the company whose management and employees keep proving over and over again they have no regard for user experience whatsoever.
From what I have seen Netflix is the most popular streaming service in terms of user experience
@@OggerFN "popular in terms of user experience?" That's word salad. UX is not a popularity contest.
Netflix certainly blows Prime out of the UX water. However Netflix's algorithm is terrible at recommending things I like. It's annoying tbh and more of a hindrance. I'd prefer just to see the categories and choose things for myself but I can only easily access this though Google searching. Lol.
Nevertheless what we are looking for here us technical excelence, not business, UX or management excellence.