Acceptance Testing | Webinar
Вставка
- Опубліковано 8 вер 2024
- Writing and maintaining a suite of Acceptance Tests that can give you a high level of confidence in the behaviour and configuration of your system is a complex task. In this talk Dave Farley will describe approaches to Acceptance Testing that allow teams to: work quickly and effectively; build excellent functional coverage for complex enterprise-scale systems; manage and maintain those tests in the face of change, and of evolution in both the codebase and the understanding of the business problem.
This talk will answer the following questions, and more: How do you fail fast? How do you make your testing scalable? How do you isolate test cases from one-another? How do you maintain a working body of tests when you radically change the interface to your system?
Bring your questions along for the Q&A at the end of the presentation.
--------------------------------------------------------------------------------------------
📖 MY NEW BOOK 'Continuous Delivery Pipelines ➡️ leanpub.com/cd...
🎓 CONTINUOUS DELIVERY ONLINE TRAINING ➡️ courses.cd.tra...
1️⃣ Get Going With CD Pipelines ➡️ courses.cd.tra...
2️⃣ Anatomy Of A Deployment Pipeline ➡️ courses.cd.tra...
3️⃣ ATDD with Executable Specifications (COMING SOON) ➡️ courses.cd.tra...
--------------------------------------------------------------------------------------------
📧 JOIN MY MAILING LIST HERE AND GET YOUR FREE 'HOW TO ASSESS YOUR CONTINUOUS DELIVERY CAPABILITIES' GUIDE ➡️ www.subscribep...
--------------------------------------------------------------------------------------------
Dave Farley's Blog ➡️ bit.ly/DaveFWe...
Dave Farley on Twitter ➡️ bit.ly/DaveFTw...
Dave Farley on LinkedIn ➡️ bit.ly/DaveF-LI
Honestly, you could learn almost everything about what it means to be a software developer from one man, by watching and listening to Dave Farley. Now if you also read his books, you're 99% there. Then if you follow his recommendations for other people that you should read/watch/listen to, you're well on your way to becoming a veteran developer, a true software engineer like Dave himself!
Master class, master class!
Thanks
Wow, world class content! Thank you
I totally get the concept of "system under test" -- I am just a little unsure what constitutes a "system" in a micro-serviced based architecture. Here is a quick scenario - A Web app with a backend layer (bff) that simply orchestrates calls to 4 other microservices. A user story with acceptance criteria may span multiple microservices. I implement a protocol driver that interacts with the web app because I want to test the system from the user's perspective... Is it wrong to have all microservices involved in that user story spun up and serving real requests? Or would you expect each service to have it's own acceptance tests and be tested completely in isolation? In other words, is the system the entire suite required to execute the user story or is it the individual parts? This is particularly relevant in self-contained systems architecture. Would love your thoughts! Thank you for all the incredible content!
1. You can write a protocol driver for the BFF, it would exercise the system under test (one of the micros) using the inputs and outputs from the BFF.
2. You can also write a protocol driver for the WebApp, in order to exercise the system under test (one of the micros) you would deploy the WebApp and the BFF and the micro, you'd hook the BFF up to the micro and the WebApp up to the BFF, along with whatever other stubs for other micros you need in place to exercise the system (e.g. you may need a stub for user authentication that returns a bogus validated session or an invalidated session etc). You shouldn't need to connect all the other micros to the BFF if you are using proper BDD and separation of concerns, whatever interactions you have with the other micros should be encapsulated as Dave said into tests of the contracts and then put into the suites for those other systems.
Perhaps what you are talking about is a system where you have a BFF that orchestrates calls to four other micros, but no matter what action you perform on the WebApp, if one of those four micros is not available, the actions blows up. In that case, you're hosed and you need to clean up your design. You should be able to place an order and see that the order was placed, for example, without needing the real-time notification system to be connected, or the email alert system to be connected, or the reporting system to be connected. You can separately test your connections to your notification system or email system etc by testing its contracts.
Key take away: Boundary and interface.
Amazing insights, thank you
Glad it was helpful!
At 57:55 you mention something along "Tests running against In-Memory-Database are not Acceptance Tests"
I think those tests are valuable and should be part of your test harness to give you fast feedback. Depending on the structure of the tests I would also call them "Acceptance Tests".
PS: The whole test harness is a mixture of tests that give you lots of confidence
Sure, I don't disagree that they may be useful, but acceptance tests, for me, are by definition running against a deployed release candidate, that is something deployed and running in the same config as it will be used in production. Other forms of testing are also valuable, but acceptance tests using a different tech for the DB aren't testing a production configuration of the system.
@@ContinuousDelivery Ahh... I see.
For me the main criteria that made a test an "acceptance test" was the collaboration part between engineers and business SME.
Thanks for the clarification on your definition!
With all these different tests one has to write, code required for testing is definitely going to exceed code being tested. Still, these tests won't catch all bugs.
Soon, we will need to write code to test the code for testing the main code.
I have never bothered to measure it, but there is certainly a similar amount of test code to production code, and probably more. But it is not about "catching all the bugs" that is the wrong focus. Unless you plan to write your code and never run it as you develop it you are testing it, so all we are talking about now is how to do that efficiently.
The trade-off here is not the time you spend writing code vs the time you spend writing tests. It is the time you spend writing tests vs the time you spend debugging problems in production.
On average, teams that work the way that I describe spend "44% more time on new features" than teams that don't. That includes the time they spend writing tests.
So writing a test is rather like deciding to diet or exercise. It is an investment in the future, rather than an instant gratification - Although once you start doing it, the "instant gratification" of KNOWING that you code works is pretty good!
This is not accademic theory, this is a practical way of working, so no, you don't have to "write tests for the tests" because that "test for the test" is built-in to the TDD process. You write a test before you write any code, run it and see it fail - that tests the test.
Sure the tests won't catch all bugs. The tests should act as specification and make sure the system behaves as expected.
Test for your tests is called Mutation Testing, and old concept that nowadays is pretty mature
How do you maintain isolation for tests that use the controllable clock. How can two tests both controlling the clock be executed at the same time?
What we did, was the bit at the end of the presentation. We allocated a dedicated version of the system for each time-travel test. So there was no sharing of the app for time-travel tests, and so no contention between them. Of course, this meant that these tests were expensive, but they tested some really interesting scenarios - we could test months or year-long scenarios (e.g. daylight saving time changes) in milliseconds using this technique.
@@ContinuousDelivery Thanks, sorry I missed that. I'll watch this part later again.
Unit tests are so called because each test is isolated from other tests. Find a way to isolate the test and you'll have your answer. Likely, you'll need to have separate instances of the clock.
Thanks for the fantastic Podcast :-)
What are your thoughts about automated real UI End-to-End Tests that will go through the whole Environment as a completion of the test methods you mentioned?
Let's say you have a complex environment with several systems and several DevOps Teams. If each team only tests its own system with isolated tests (with API and UI Tests), will that be enough to make sure that End-to-End Processes are still working, so that no higher level Tests are needed?
What would you suggest for a strategy in mixed environments where DevOps systems have interfaces with legacy systems and different delivery models?
In general I think we should get as close as we can to testing what goes into production. So I prefer, at some point to test the UI, at least enough to show that the pieces are wired together corectly and so work. That is not enough to test all UI features, so I prefer that UI components are also unit tested.
For me the scope of evaluation is an "independentlt deployable unit of software" so SW that you are happy to deploy alone without regression testing with other parts.
I explore some of these ideas in other videos, here are a couple:
Acceptance Testing: ua-cam.com/video/JDD5EEJgpHU/v-deo.html
Testing at the edges of systems: ua-cam.com/video/ESHn53myB88/v-deo.html
When you use the generated names, how do you refer to that name later in the test. Do you make the created instruction return the generated name ? If so, isn't it less readable and more cumbersome to understand than plain string, when the test fails ?
The test infrastructure maintains the map in the instance state for that execution. So you can call the book "Continuous Delivery" and continue to refer to it as such throughout the test, and the DSL will always map it to the real name that the app uses.
The only confusing bit is when you are debugging a test run, then you will see the real names, rather than the alias that the test uses. For that reason I prefer to not be completely random in the creation of the real names, I like to retain all or part of the name that the test specified so that I can match them up. So my example of "Continuous Delivery 1234" and "Continuous Delivery 6789" for "Continuous Delivery" was fairly representative of what I'd actually do. The names don't need to be globally unique, only unique within any given run of the acceptance test suite of tests.
@@ContinuousDelivery Make sense. Maybe you have a video on acceptance test infra, it would help to fill the gap between principles and implementation.
Well there will certainly be some of that in the Acceptance Testing training course :)
Hi Dave. Thanks a lot for the webinar. Could you share a bit how junit was used to run the tests conditionally and to inject different protocol drivers? I assume someone wrote a junit extension to do this, but I wanted to confirm it.
Also can you share a bit how the data was created behind the scenes? I find this aspects one of the most complicated ones, which also makes the test more difficult to understand.
It depends what you want to do. If you want to support running multiple versions of the test with different protocol drivers, you probably do need to do that at the level of a custom test runner, that is what we did. If you just want to select the single one that makes sense in the context of a particular test, you can do that as part of the DSL implementation at the point where you are about to hand over control to the ProtocolDriver. I generally implement this as a composite pattern, with my PD implementation represented as just another PD, which actually contains a map, or dictionary, of PDs key'd by the method annotation - look up the annotation, find the key, return the correct driver.
@@ContinuousDelivery Thanks a lot Dave. I need to get a bunch of stickies and my thinking hat :).
Please tell us how you got to be a cd expert.
Well I suppose it depends on what you mean. In part I am seen as a CD expert because I wrote the book "Continuous Delivery" along with my friend Jez Humble, but probably the more accurate answer is that I learned about what worked in SW dev by doing lots and lots of SW dev and doing things wrong and finding out what worked better. I started off doing CI (though we didn't call it that then) in the early 1990's and learned about TDD from Kent Beck in the late 1990's.
I led several of the projects that are now seen as projects where CD was "invented".
Is the book available in India as well?
yup I just bought it on amazon
I appreciate the work you do Dave, but line 2 at 22:17 is making my nose bleed
Completely unrelated question: do you speak Japanese? I see a 継続的デリバリー book on the shelf...
Well, I wrote the Continuous Delivery book, so that is one of my collection of translated versions, so I am afraid that I don't speak Japanese, except for konichiwa!
@@ContinuousDelivery Ahh, I see. I knew about you being the author, but I thought you also were one of the people crazy enough to study Japanese hahaha.
Anyway, thanks for the great content! This is by far the best software engineering channel on UA-cam!
@@glhrmcosta91 Thank you