Quality Assurance in Agile Software
Вставка
- Опубліковано 15 тра 2024
- What is the role of QA in high-quality software development? Is this about quality assurance or quality control? QA is one of the roles that changes significantly in the transition from traditional software development to high-performance agile software development. It’s not all about QA testing or QA as gatekeepers. So what is the role of quality assurance in a modern hyper-agile continuous delivery team?
Agile software development is really about moving fast and learning things, speed of feedback is of the essence, how do QA professionals fit into organisations like this, and how do they contribute most to helping to deliver high-quality software quickly and efficiently?
In this episode, Dave Farley explores the role of QA in modern agile teams and explores the move in some detail from gatekeepers to quality experts and ideas like continuous testing and QA as trusted advisors.
_____________________________________________________
📚 BOOKS:
🚨 MY NEW BOOK! 👉 📖 Dave’s NEW BOOK "Modern Software Engineering" is now available here ➡️ amzn.to/3DwdwT3
In this book, Dave brings together his ideas and proven techniques to describe a durable, coherent and foundational approach to effective software development, for programmers, managers and technical leads, at all levels of experience.
📖 "Continuous Delivery Pipelines" by Dave Farley
paperback ➡️ amzn.to/3gIULlA
ebook version ➡️ leanpub.com/cd-pipelines
📖 The original, award-winning "Continuous Delivery" book by Dave Farley and Jez Humble ➡️ amzn.to/2WxRYmx
NOTE: If you click on one of the Amazon Affiliate links and buy the book, Continuous Delivery Ltd. will get a small fee for the recommendation with NO increase in cost to you.
-------------------------------------------------------------------------------------
Also from Dave:
🎓 CD TRAINING COURSES
If you want to learn Continuous Delivery and DevOps skills, check out Dave Farley's courses
➡️ bit.ly/DFTraining
📧 JOIN CD MAIL LIST 📧
Keep up to date with the latest discussions, free "How To..." guides, events, online courses and exclusive offers. ➡️ bit.ly/MailListCD
-------------------------------------------------------------------------------------
CHANNEL SPONSORS:
Linode offers Linux virtual machines with global infrastructure and simple capped pricing. No surprise bills, no lock-in, and the same price across every data center. See if Linode works for you with a $100 60-day credit by signing up HERE ➡️ linode.com/cd
Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ bit.ly/3ASy8n0
Harness helps engineers and developers simplify and scale CI/CD, Feature Flags and Cloud Cost Management with an AI-powered platform for software delivery. ➡️ bit.ly/3Cfx3qI
Octopus are the makers of Octopus Deploy the single place for your team to manage releases, automate deployments, and automate the runbooks that keep your software operating. ➡️ octopus.com/
SpecFlow Behavior Driven Development for .NET SpecFlow helps teams bind automation to feature files and share the resulting examples as Living Documentation across the team and stakeholders. ➡️ go.specflow.org/dave_farley - Наука та технологія
I came across your channel last year, and it continually amazes my just how in sync we are _(i've been developing software for ~3 decades)._ It feels like you're speaking my mind. I've been *really* needing that; sometimes, with certain teams, I've felt as if I were seeking something that's too idealized and unrealistic. Now I feel vindicated. Thanx a bunch!
Glad to help. Thanks for your support.
I think a lot of us have this feeling often. It's a matter of perseverance and demonstrating the benefits to an ever growing group.
@@girian You're most definitely on point. In my career, however, I had to make the choice whether to drive myself absolutely insane within an undisciplined team or to go it alone... driving myself absolutely insane on a years long journey, building a creative and development platform all by me onesies. All things considered, it looks like I made the sanest choice. Later this year when I release, tho, we'll see if I'm right. God, I hope I'm right.........
You could say this channel has been _continuously delivering_ quality content
This was a really great insight and as a QA Engineer, I agree it is always good to have kickoffs with devs before they start development and help them give relevant test cases and discuss potential ripple effects on any existing features. Thanks!
My wife works at a pharmaceutical company in QA. I am a semi-retired software engineer. Your videos interest both of us!
Dave, i completely agree with this shift left approach, thanks for evangelizing it!
Thanks for the video. I do agree on several points of the video. The 'old approach' to QA is ingrained in most teams. The 'old approach' tends to be dictated by either a group manager or CTO as they have a set way of doing things. I have sold a few teams on the new approach with some success. To IT professionals that want to use the new way of QA (as the video suggests) I strongly suggest pointed and deliberate questions in the interviews.
Thank you Dave, I very much appreciate the way you share your knowledge and experience with the world.
A critical note though:
In the video, the role of QA seems to be restricted to testing, during the development phase.
However, QA is more than testing. QA is also about motivating the team to do some form of root cause analysis, which isn't necessarily a time-consuming formal meeting, but can be an informal chat between QA and a Dev. Update the bug report (if there is one) with the cause and lessons learned. Maybe the bug learned us that we should refactor some parts of the code, or make the logging more readable and valuable.
QA is also about reporting on the number and severity of bugs that make it to production (preferably in relation to the number of code changes). Are the test efforts paying off?
I manage qa resources across our org and I am in 100% agreement. The challenge is getting all the different teams in alignment across the org.
Thanks for this. This is pretty much exactly how I've been thinking the process should go. In general, it's another problem of people falling into their own isolated silos, to emerge only when they think they've finished. Perhaps unfortunately, some industry standards require manual testing to be done before release, with a release candidate, with meticulous test plans and reporting. Most projects do not need that, and even if they do, those test plans are often best done together rather than making them someone else's problem.
Dave you are spot on. I would dare to say 95% + of QA organizations are tied to the same QA practices from decades ago. 3 Amigos embracing BDD has elevated the QA role and not thought of as a "QA gatekeeper".
Great video Dave, very useful.
Thanks!
Excellent video, thanks Dave.
Your videos are the sugar on top of your books.
Such an amazing presentation. Thanks a lot.
Amazing... thanks again Dave
So it is not only Continuous Delivery, also it is Continuous Testing. Good point.
I will get the book right now; one, because I am so curious on getting deep into the Farley's thoughts on this approach and second because It's the least thing I can do for supporting this great work. Yea, probably the majority of the organizations are still far from this approach, but every QA and DEV that works with QA professionals watching at this video should do it's part to make it real and take the collaboration between teams and efficiency to the next level.
Thanks 😎
Great video, your ability to articulate and explain concepts simply and clearly is next to none. I look forward to every QA testing video you make and I'm becoming a huge fan 🙂
I wrote a blog post on this recently as it seems most testing, manual or automated, is just asserting underlying code functionality with a binary pass or fail result. Fine for regression testing, but where is the testing of the value described in the user story? QA testers also test this by investigating and telling counter user stories of threatened value and harm. This can even be done during story mapping before any coding or even designs have been done, so the three amigos/amigas and manual testing points resonates with me really well.
Contentious testing throughout the SDLC: For every development or PO activity in devops, there's an equal but opposite QA testing activity running in parallel.
I work as a software tester more or less in a way you described. I say less because of a number of reasons:
1. It's a 17 minute video about quality, yet there was not one hint at what quality means. I find this problematic because if we don't agree on some shared definition, it might end up devs thinking quality means quality of their code, whereas testers typically understand quality as value to our customer. These are not the same thing and people will work differently based on what they believe quality is.
2. I'm still surprised that there seems to be a notion that testing equals demonstration that the software can sometimes work. It's maybe part of testing, and not even a very important one. Testing answers the question: "What are the problems with the product?" That's not the same as checking the product can sometimes work.
3. "Software testing proves the existence of bugs not their absence." Dijkstra said that many many years ago, yet we still believe that our job is to assure the software is without bugs. The name quality assurance is problematic for this very reason, in addition to the fact that people, even QA people, many times have no idea what quality means and what they are assuring.
4. There is no manual testing, just like there is no manual development. What would manual development even mean? How about manual doctors? Are pilots manual or automated ones when most of the flights are handled by a system? Why is it that testing seems to be the only role that's strictly divided into manual and automation? Does it mean that manual testers doesn't use tools to make themselves more powerful testers? That's ridiculous. I find it more useful to devide testing into attended and unattended testing based on whether or not a human is involved in performing testing.
Very insightful. Thank you!
Thank you, Dave. I have a question for you :-) what would you say if the product being built is a financial aggregator? Production incidents are expensive (sometimes extremely expensive) and require a lot of additional formalities like informing company's partners about failed transactions, etc. Having early feedback sounds fantastic but comes with a high price in this scenario. Do you think there is still a way to implement continuous delivery?
your advice are very good, it helps me a lot, I start to implement them one by one. It make life gradually easier.
If I may suggest topic for video it would be about how to with steps apply your advices. Many firms still live in 'dark ages' and it can be hard for them to start to implement good practices.
This channel is very much a blessing
Thank you!
Good video.
There are ideas here I believe deserve expansion and more discussion, but this is a strong backbone video to base them on. Especially interesting is how to address the big test problems that remain even after the 'check it as we create it' has reduced the defect probability such a large amount. Maybe you have something queued up, but I believe a partner video with some testing experts would be good.
Great and simple explanation!
My biggest challenge with CD pipelines and QA is where/how to include automated regressions into the pipeline.
Doing multiple atomic changes in a CD pipeline is at some point easy and fast, but when full regressions are part of the pipeline as a quality gate, it start slowing down the process a lot.
"Full regressions" are built in to the specification of the pipeline as I define it. So they are included by default, and aren't an add-on.
Regression testing in CD is fully automated, manual testing is reserved for only exploratory testing, and the most effective strategy is for regression testing to be integrated into the development process - we start new pieces of work by writing an Acceptance test that specifies what our users want of our system.
The problem then is optimising them so that you get results fast, and if your project is big, scaling up the hardware to run it all on - so cost.
Google do this as MASSIVE scale, at LMAX where we built one of the worlds fastest financial exchanges, we did full regression for our entire enterprise system, comprising something like 70k-80k automated tests and got results in under 1 hour.
THANK YOU!!! This is very close to how things worked in my first QA position and I loved it. My next job had wildly different ideas and it hurt my soul lol. I'm onto my latest company which is in a state that I will be able to implement this thinking upfront to hopefully avoid all the pain points I'm currently suffering with the mini-waterfalls hahaha
We change the world a small step at a time 😁😎 Good luck!
Manual testing also has the benefit of upholding knowledge of the system(s). These manual testing sessions might be the only time developers interact with certain parts of the system and learn how the are supposed to work. If we stopped doing manual regression testing we would soon loose knowledge about what the system does and how it works.
QA's job is to _assure quality._ So as long as the quality of the code is good, QA is doing a great job.
QA should work on setting standards for e.g. test coverage and static code quality rules, and reviewing the tests that are run or writing those tests before the code.
Great video. I've watched it, because it didn't have this click bait title and thumbnail, like some recent videos, which I skipped. Great job 👍
Glad you liked it!
Hi Dave, thanks for your video. It speaks to some challenges that I’m facing in my organization. I have a question though. What role does manual QA testing have when your software does not have a user interface? Do usability and exploratory testing apply here? Thanks!
Another great video Dave! I have greatly enjoyed your weekly content so far and I share many of your viewpoints on what good software engineering should look like. With every video, I long to be part of an organization where people are driven to do things right. Unfortunately this seems so rare, and even if we want to use our newly acquired insights to bring about change, we often run against political walls and resistance in the organizations we work for. I say "we", because in the comment section I frequently read that others struggle with the same problem. Perhaps this could make a topic for a future video, how to drive change in an organization and convince our peers and managers that we can do better by following advice like yours?
Funnily enough, I have a video on that topic in development, keep watching - and thanks for the suggestion 😉
I'd suggest the approach of finding like-minded individuals, and going for it. Don't wait for permission, just ask for forgiveness. ;)
Great presentation!
Your insights will be very instructive in reviewing our Q/A discipline.
Do you have any thoughts on fusing the roles of scrum master and QA? -- they way how you presented the QA responsibilities, I felt like it could fit the job of a scrum master?
A thing that is often neglected is test maintenance. Tests break as software is refactored and underlying technology and infrastructures change. Ideally, they would be fixed or retired the moment a breakage is seen and that would be a joint effort of Dev and QA halting development until fixed. But often development moves on while QA struggles to keep the tests running. As time goes on, the amount of code typically increases and so does the amount of tests and test breakages, but the size of Dev and QA teams doesn't keep up. So if not handled carefully and kept in check, test breakages can overwhelm a team. There needs to be clear understanding, that broken tests halt all development.
This seems like an essay on why test driven development is a must on CI/CD teams.
Nearly everything I do is that. 😉
The one hill I will die on: Stop having QA departments, stop having QA teams and please, stop calling testers "QAs". Heck, ban the word QA if needed.
My reasoning is simple - Having a separate group/role called "Quality Assurance" entrenches the belief that quality is the responsibility of a separate specialist group and that they can "inspect quality in".
Some will interpret this as "you just dislike testers". Nothing could be further from the truth. Testers are awesome and are usually paid to little and treated as second class citizens, but testers are not QAs, nor are they in charge of quality or "approve" of changes. Testers brings a unique perspective and skillset to the team and provides input and feedback so that the entire team can make informed decisions about the next steps.
As someone else has mentioned elsewhere - "DevOps is not a team/role, it's a feature of the team" and just so is Quality Assurance a feature of the organization as well.
I had a team where the devs refused to “cross over” into automation or QA minded approach. Instead, we had QA/QE that wrote redundant automaton test scripts across every story the devs coded ( which they had written unit tests for). It was checking a box and as the product owner it was maddening to get pushback from QA management when I wanted the team to change how they approached it to a method similar to what Dave is saying here.
absolutely incredible
@13:35 Are the executable specifications you describe at this stage something that could be filled in by developers as unit tests, or are these always going to be integration tests?
As a QA tester with focus on manual testing, I find this video really interesting, but also: quite disheartening.
We are doing "CD/CI" but watching this video. I'm bloody Gandalf there, stemming the tide because we are stuck in a limbo between the two worlds of how to do things.
It is a big step, but it is the right one I think, good luck.
5:56, throughout projects I work, I use to have "known issues/new ideas/refactoring tasks/reports docs", even in intermediate releasable states. I'm used to test every change, but once detected, I don't necessarily solve the bug, taking note instead.
15:10, what is this?
16:44, unless the coder stops everything to catch the bug, I think is better take note from the bug. Plus, there could be 2 or more bugs detected roughly at the same time, or the 2nd bug araising before the 1st is cleaned.
Very powerful. Thanks for sharing your insight. When you take this approach, how does the delivery timeline change? If you have a client who says they want X, Y, and Z and you have a set timeframe and possibly budget in which to create that, how do you keep the project moving forward without getting stuck in a cycle of developed, test, fix, test, develop, test, fix, test especially when you may have a sprint with an agreed upon set of tasks? Say you have a feature and you spend twice as much time as you planned to get it working correctly. You now have that much less time for another task that was part of the sprint. This causes you to burn up instead of down.
I don't know in terms of output, because it is hard to measure output in the way that you characterise it. Subjectively, you have it the wrong way around, this is significantly the productive way of working. The data on throughput, different to output because in CD we work in smaller steps, is interesting and certainly related to output, teams with high scores on both stability (quality) and throughput (efficiency) spend 44% more time on new features.
I once worked in a company that worked this way. We hired a senior manager from the finance industry, after about 3 months he was chatting with the CEO and said, "I have to be careful what I ask for here. I am used to asking for lots of things and never getting them, here ask for something and it is in production the following week. So now I need to ask for the right things". A few years later he left and started a (now very successful) Fintech startup and made sure that it worked this way.
most important for manual qa is noticing something that was added to the system that doesn't belong. Automation cannot really look for things that dont exist yet
Test every commit, in fact, where possible, do live testing. Also do pair programming, so that you have live review without downtime.
...and cut the amount of work you do in half.
Such an important video Dave, answers so many of the neigh Sayers on how to do things well!
Great video I would be interested to understand if the automated test the QAs write are seperate to the code? They are in my experience but maybe they should be integrated within CI pipeline.
No, I think it easiest when the tests and the code live together in the same repo. Then it is clear that this version of this test works with this version of the code.
Keep the good stuff coming. Technically savvy QA guy here. :) Do you have a recommended QA-to-DEV ratio, in order to properly man all 3 Amigos meetings?
I think it varies a lot, depending on the nature of the organisation and the system. On the teams I worked on, we usually aimed for something like 4 to 1 (dev to qa) sometimes 6:1
@@ContinuousDelivery Thanks!
We switched to 'agile' but the organisation is anything but agile. They just made things worse and still work with waterfalls. even introducing CI/CD seems difficult. I've seen this work in other projects but in this case it's hopeless. I can't change the organization nor do I see it change in the future. But sometimes you must move on
It is so true that the organization’s prevailing ways of doing IT are a huge roadblock. If the organization is designed to operate in a waterfall “project mode”, then it’s going to be tough to make this sort of transition. The good news is there are growing legions of people who want to work differently in better ways, so hopefully more and more organizations will get the leadership that is needed for them to make necessary changes -or otherwise be out out of business by more efficient competitors. Hopefully not just bought out and subsumed…
Thanks for the video. Do you think the QA Automation role will disappear?
I don't know, I think that QA as "after the fact manual gatekeepers" should disappear.
Great video, I agree with most points there.
In my experience, what makes things go complicated, is that there are always tests that just inevitably take long to finish. See, when designing and implementing tests, you want them to reflect the real world with some level of accuracy. For example, if your customers are using hardware XYZ, you want to ensure that your test plan includes tests ran on hardware XYZ, but if the hardware is rare and/or expensive (and hard to simulate) your test pipeline will fail to scale: you simply won't be able to run every test every day.
This typically affects high level system tests (sometimes referred to as end-to-end tests), which will necessarily span almost whole of the complexity of whatever your system is, so I think this issue will affect more complex systems much more, and it helps a great deal if you can split your system into components and develop them independently, but then again, confidence in parts does not always translate to confidence in the whole -- so you will want to run the whole thing anyway.
The problem is made even worse when the hardware, or whatever provisioning system you are using, brings own noise into the system for some reason, so even if your test can totally finish twice a day including waiting in queue, one in 10 results will be useless and will require manual re-runs, etc.. because some freaking network card keeps dying or something. Unfortunately issues have been regular in most places I've worked over my ~14 years carreer in QA/QE.
Testing is about exercising control over the variables, so how do you eliminate the variables that trip you up? If your hardware is flaky, get better hardware or find a way to take it out of the loop in the test. If the hardware is part of your offering, then the tests are telling you to design better hardware!
I see many firms and teams accept as some truth that stuff is flaky and so untestable, when what it really means is that the code (or system) is low quality. Computers are deterministic. Where they are not, it is usually as a result of poor design, often poor isolation of concurrency.They are complex, but they are deterministic, and we can use that. There is more to it than this, but I think that this is a good starting point.
@@ContinuousDelivery Totally agree. Last 8 years I've been working for Red Hat, (testing RHEL) but in many ways that is different situations than other companies I've been in. We don't design the hardware, the hardware is often extremely expensive (mainframes..) and production deployments can be huge (ie. we can't say "sorry but you need to get better hardware"). In the meantime what we are delivering is pretty complex system, which can be, and is split and tested separately, but there are limits to that (eg. installations or upgrades). (We also don't design the software, for the most part, but that's a whole different story..)
That said, as QA, we've been always trying not to fall for the trap of accepting the flakiness, instead look for a way to improve it. It is an exercise in technical leadership, though, and not an easy one, but we have to keep what you said in mind.
@@ContinuousDelivery A Cambridge dictionary definition of testing is "The process of using or trying something to see if it works, is suitable, obeys the rules, etc." There are many different things that could come under that definition depending on the context.
Any software Tester with a foundation certificate knows that the earliest and cheapest opportunity to identify a fault is doing static analysis testing. Testing the wrtten requirements. Few orgs do this properly. Further, most defects are found in touchpoints between features so testing granular units does not mean less defects. The impact of small changes can also be high and therefore demand significant and time consuming regression testing of the complete solution. Automated testing of granular code might be quick but maintaining all impacted automated regression tests to accommodate continuous small changes is a massive challenge. manual exploratory and risk based testing would be more effective in keeping up with continuous development and integration. I use QA check-lists between different types and phases of development and testing for risk mitigation and inform a decision to proceed to the next phase of development. Different test cycles have different test objectives and apply different test techniques accordingly. It's not simply a case of testing.
I've been working in a QA Team specialized in PCS (Performance, Capacity & Scalability) testing for 5 years.
Our whole process was entirely manual and as you said after-the-fact when I joined the team. Today our activity is much more automated but still not part of CD since CD is a relatively new concept in the company I work in.
Today my team and I are challenged to implement PCS tests into the CD process which apparently means that our QA team will have to deliver a product capable of providing not only the tests (scripts/tools) but also the whole validation mecanism and test infrastructure for load injection integrated into CD.
My understanding is that since CD is very new in my company we only have basic validation executed in pipelines such as smoke tests and unit tests.
Now, I am all for the approach to test as fast as possible and as soon as possible and that CD is the solution for that. However, it is still not clear to me what is the role of QA, really.
Some things that are not clear to me
> What is the role/scope of QA in CD ? Sorry but I haven't gotten a concrete answer from the video.
> Do we deliver tests to be consumed by CD ?
> Or are we developpers of our own QA Testing Product (itself developped in CD) consumed by CD pipelines
> Today I'm being asked to become a Product Owner of my Testing Service, to be integrated into CD on top of our traditional QA activities which sounds like a very ambitious endevour to undertake.
> How to take into account the complexity of tests ?
> The tests that we have today are complete end-to-end user scenarios that often solicit many services at once via web APIs, that we auto-generate on a given test environment to ensure that the APIs we call are up to date (API discovery by sniffing while we run the scenario interactively via Selenium/Sikuli...etc + auto-parameterization for logic and handling different object IDs, user sessions, security...etc.). The generated test plan is then used to run what we call a PCS Test Campaign composed of varying numbers of simulated concurrent users (i.e. 1 virtual user, 100, 1000, 2000 vitual users). Each test is ran for approximately 10 minutes so we can generate a lot of samples and this also acts in a way as robustness tests (for problems that may arise after extended usage [memory leaks, performance degradation over time,...etc.]). The scenarios can be as simple as navigating through different UIs only reading information from remote services, but also creating and manipulating data, and making virtual users interact with each other via Real-time messaging and audio-video call services. So each Service has 1 to n PCS scenarios and each test campaign for each scenario takes about 1h to run. There are a lot of good reasons and Best Practices that justify why we do these tests the way we do them but my feeling is that, these kinds of tests, from a complexity and time consumption point of view are not fit for CD. I guess if we want to implement PCS tests in CD we will need to take a different approach where we do more targeted and simple tests that run for a shorter amount of time which would mostly show the most blatant and immediate performance issues. So clearly QA would still have value outside of CI/CD for more advanced tests.
Any thoughts on all that?
Sorry for the long comment.
Thank you in advance.
Watch more of my videos on testing strategy and approach to get a fuller answer. Testing needs to drive development, so QA's can't own it. The role of QA is to become an advisor, expert, on thinking about how to create better quality products. So advising on test strategies, not implementing them (they may help if they have the skills). I think it an anti-pattern if anything QA does takes away responsibility for quality from dev teams. QA as part of a dev team can reinforce the quality perspective, but it is not a QA role to absolve other people from taking responsibility for the quality of their work. So QA as teacher, quality coach, helper, not tester or gatekeeper. It's the team's responsibility to test, and QA may help.
For the sprint kickoff (three amigos), what about UX designers and Operations?
In front end implementation, many teams have UX involved in design, often QA left out of process.
For back end components, even using cloud resources, often operations is not part of the process, and it fails spectacularly due to false assumptions that would have been spotted easily by ops.
Same idea really, you need the skills necessary to do a good job available at the point of creation. Doing work ahead of time is a problem, what if that work is wrong, and doing it too late is a problem, as this video describes. The nature of the work doesn't change these facts. So try and get ops thinking into dev teams, that was part of what DevOps was all about, and try and get UX design focused on generalities and guidelines, guard-rails rather than detailed screen designs, and staff and educate development teams to be able to implement their own designs within the guard-rails.
Having been in a QA role in the past, it just has no place in modern CI/CD. QA engineers have to diversify their skillset, become polymats of CI, dev and testing for the purpose of shifting testing left.
Dave, in your experience when you had QA engineers actually contributing to writing the tests, what was the ratio of devs to QA on a given team? I wonder how QA would keep up with a team producing changes throughout the day if they were the ones to actually write the tests.
The QA team often, not always, wrote the top level script for acceptance tests, this was written as BDD scenarios in the language of the problem domain. So they were relatively easy to write. Devs wrote, and maintained everything else about tests, unit tests, performance tests, as well as the plumbing that makes it easy for anyone, including QAs, to write acceptance tests.
The usual ration on teams in our org was 1:QA/4-6:DEV but it is important to re-state, testing was NOT a QA's responsibility. Testing is the team's responsibility, and some parts of it, sometimes a QA could help with. If the QA was busy when the team needed acceptance tests, someone else on the team wrote acceptance tests.
@@ContinuousDelivery: Gotcha - So, In that scenario, the devs would write the DSL, and the QA team would write the test case layer. In that case, I could also see a QA engineer possibly writing a protocol driver implementation for the UI. Just seems that is fitting for their skillset. Again, only in the scenario where QA was assisting/complimenting the team on quality, not owning.
@@jordanschwartz6871 No quite. Devs owned the tests overall, all the layers, and maintained them. Anyone, including QA, could write test cases (specifications), if the QA needed features that weren't in the DSL yet, they could invent the language to express the idea in the test case. Then devs would add the new support into the DSL, taking the language in the test case as a kind of requirement. They could change the syntax, to be more in-line with the DSL or more abstract if they wanted to, usually in discussion with the QA who wrote it. The dev team always wrote the protocol drivers, and maintained them. These were the code that know about how the SW, that devs were changing, worked, so definitely the dev's responsibility to maintain.
But in a continuous delivery (or especially continuous deployment) world, in what environment are these exploratory manual tests supposed to be done? If it is one that is pointed to your main branch you'd have to feature-flag everything if you wanted to ensure this testing happens before the software is released.
In Continuous Integration, trunk-based development is essential. So yes, the version that is being tested is from the main branch. The point here is that it is not QA's job to gate-keep, test automation determines of the code works and where we can release or not, QA is about other forms of quality. Sure if they find something that is really s stop sign, they can ask th4e devs to fix it or pull the change, but this almost never happens. Most of the kind of bugs that QA people find are more subjective, and you can live with them for the short time it takes to fix them and get the fix into production.
@@ContinuousDelivery I get your point about test automation being our gate, not manual qa, but you make the point that QA is involved from the beginning, and seeing development as its happening. This means the QA need an environment to do that against. From the book (page 7) "if testers have been involved in the process up to this point, they have tested the system on development machines." For a microservice architecture that is delivering a web application, this means they and devs have an integrated environment locally, or runnable in a branch remotely. Those types of environment are very hard to setup and maintain, so often we need to get testers involved after the software is in a production like integrated environment (aka staging). I think this is described as an antipattern in the book.
What does one do?
- Invest heavily in creating(aka staging) that can be run by devs and qa while code is *not* yet in the main branch. this could look like local environments, or ephemeral/review environments
- feature flag changes
- provide ways to temporarily deploy changes to an staging environment that aren't yet in main
i think we want fewer environments not more (see ebay's approach to staging here -tech.ebayinc.com/engineering/the-staging-dichotomy-part-one/). , but this is a sticking point for us.
Might be a cool video: environments: local, review/ephemeral, staging. tons of angles to it.
@@thigmotrope We can have as many environments as we like with TBD. Because now we have a clear stream of versions, each one, in CD land, releasable. So all we need now, is the ability to easily, reliably, create these environments. So we automate all of that too. Where I worked, a QA could pick a free test env, pick a release-candidate, push a button and deploy that release candidate to that environment. Simple.
I think most of your points are relevant for very simple applications and setups, but may be more challenging in a real world setup. TDD will help you realize that your software (the code) does as the programmer intended, but not necessarily what is needed for the application. The programmer may have misunderstood the requirements, and the tests she writes won't reveal that without scrutiny from others.
Also, there are many aspects of quality to implementing a requirement, and many of these aspects are not included in the requirements descriptions. For instance: is it legal to implement it the way it's done? Does it perform? Does it scale sufficiently? Does it work with the software from the other teams and does it respect the architectural requirements? These aspects can be vvery hard to test during a development cycle.
Finally, the developers may not be sufficiently skilled, and that may not be so easy to find and hire developers that are.
Unskilled developers or misunderstood requirements is where pair programming and and QA come in. All of your missing requirements, it sounds like you're missing a good BA and/or Product Owner to deliver you well-groomed user stories.
I have spent most of my career building fairly complex systems, so no, this isn’t just for “very simple applications”.
What about executing regression tests after every small change in the code? Should it be full regression? Sometimes it takes hours.
Then make better regression tests 😉
@@ContinuousDelivery True.
It is not uncommon unfortunately. In your case no process is gonna help, you need to address this issue first, imo 15 minutes tops. I've worked with projects where pipeline would take up to an hour and fail sometimes for no reason just to pass when retried, that was a great waste of time. Nowadays I take time to make sure tests are efficient.
The other thing you can do is target what tests offer the most value in a regression
Or run the regression on a nightly
Hours for CI pipeline is too long in most contexts, you might have too many tests or tests that are inefficient. Test automation is just another piece of software, what tests can you remove that are low risk? What tests can you refactor to run more quickly? I've heard people mention targets of 20 mins which CI tests must not take longer, though as you say, you can have longer running test libraries that run less frequently for sure. So like only core tests that run on every small change and extended tests that run every day perhaps?
dave - how would you test ui 😟?
My preference is to maximise the amount of code that I can test easily with unit testing, so that interactions with external inputs and outputs is isolated, abstracted and minimised in terms of code and complexity. Then you deal with these "edges" differently. I prefer to reinforce my unit-testing with what I call automated "Acceptance Testing" this evaluates high-level scenarios, through the UI where applicable, but doesn't aim to exhaustively test the UI. The logic behind it though is fully tested with unit testing. Depending on the nature of the system, this approach combined with good exploratory (manual) testing is enough. I have some friends who do done some other stuff around automating the validation of specific UI states, effectively taking a snapshot, under automated test-control, and verifying the results in the next test run - approval testing for UIs. The clever bit is in automating the handling of failures. My friend Gojko Adzic has written some tools (open source) that will show you a "before and after" comparison of the snapshot when a test fails, you click on the one that is correct and the test remembers for future runs. So if the change made the test fail, and you agree the test should fail, it stops the release, if you think the difference is acceptable, the test tools "remember" the new picture and use that in future.
In general I am suspicious of trying to be too precise in testing UIs because they change all the time. Gojko's testing is probably as good as you can get, but it still needs human support to check releasability. For most systems, I don't think that you need that much precision, so a more behavioural approach to testing works fine, backed up by manual exploratory testing to just verify that stuff still "looks ok".
Does all what you Dave have said here imply/assume that a Manual Tester role should/ought to be fulfilled by a UX Designer?
Not really, but exploratory manual testing is largely about usability, you just don't need to be a UX expert to do that.
What do you do if tests run for a couple of hours? It is not efficient to test often in this case
Write better tests, or find better ways to run them in parallel, depending on why they are so slow.
@@ContinuousDelivery I would really like to see you re-writing them. But no acceleration would mean no pay :D
Seriously, it's an extensive system
@@toooldtobejunior There are ways to do it fairly quickly, slow tests are a cost, as you have already pointed out. They prevent you from doing a good job, because you can't get feedback fast enough. So at some point, the investment in improving them is worth it.
@@ContinuousDelivery I pointed out that fast tests would be a cost :) It would be a hundred thousand a year only to keep the test infrastructure alive all time.
@@ContinuousDelivery Sorry but it just looks unrealistic what you say about CD.
I don't really get how QA can catch issues of the in-progress tasks when there is a code review process and nothing gets deployed on the staging environment until the PR gets merged.
It's simple really, don't work that way. You you don't need a PR if you have the constant review of pair programming, for example. You don't need a separate branch if you adopt Trunk Based Development (as another example). There are other ways of working, and the data says that they are better than the way that you describe.
@@ContinuousDelivery Thanks for the reply. This sounds nice, but I can't project this onto reality. Pair programming is tricky when people have different levels, skillsets go on PTOs/Sick-leaves. And it usually ends up being one person mentoring another. It also doubles the implementation cost while not guaranteeing the quality. It is especially hard to justify when the budget is tight. And I've never seen Trunk Based Development working on a big project. Sometimes you simply can't merge big features partially without the integration step, and the process becomes a chimera that is a nightmare to manage. Maybe I'm just illiterate and need to read your book.
@@MTandi Well it's not theoretical, I led a team that worked exactly this way, building one of the world's highest performance financial exchanges. I know it sounds weird if you haven't worked this way before, but it works, it works in technically difficult, regulated industries, and it works better than the alternatives. None of this is just my opinion, this is fact based on real world experience of teams all around the world, doing interesting, difficult things. For example, this is how SpaceX develop software - I don't know about the pairing for them.
@@ContinuousDelivery Thanks, I will dig into this more as it's very appealing.
An analogy for this
Is software meant to be like taking an exam?
Are they testing your understanding of a topic or are you both working towards the same goal
To often in software, QA were 'catching' developers out, the problem with that is, you know your software is going to fail, so why even bother
It's like saying you know your going to fail the exam, so why bother taking it?
These are anitquated ideas. Starting from the 1980's these ideas were debunked by testers. Anyone who has these ideas clearly shows a complete ignorance of testing/QA (this includes Dave Farley). Research Kaner, developsense, satisfice and geraldmweinberg.
@@nilanjenator you have some level of exploratory testing but the majority has to be done up front
When you say that often QA write tests, those tests aren’t unit tests right? Devs right that because of TDD right!?
Yes, QA don’t write unit tests.
@@ContinuousDelivery Yes, but QA will write integration, system, and perhaps manual tests. Non-functional requirements (such as accessibility and performance) tend to be overlooked by developers; therefore, QA tends to still give value in those areas.
@@queenstownswords I don't think he is questioning the value of QA, just their position as gatekeepers, and that developers should also assume the burden for the quality of the end product. But this involves a shift in mentality for the organization, because TDD is not free and can't be fixed by just hiring more people.
@@cristi724 Hello Cristian. I do agree that QA as gatekeepers is very poor. QA as gatekeepers creates an 'us vs them' dynamic in teams between QA and developers that can get unhealthy.
I also agree that developers owning quality is a must.
From experience, the hard part of the approach (prescribed in the video) is to get buy in from management.
So what's the difference between unit tests and integration tests and why QA should write the latter and not the former? In the end, an integration test for one system is a unit test for another, bigger system. What makes QA uniquiely positioned to write integration tests, and why developers are not good for that? Is the argument that developers are focused more on internals and "the how", whereas QA are more aware of the system's surroundings? if yes, why? Or is this because QA is assumed to be not qualified enough to use APIs and to write unit tests? What is the value of a separate QA person, if not to challenge the developers, and be critical to their understanding and implementation? And if that's their main value, why can't it be invested into unit tests as well?
QA, or independent testers, often have a much higher understanding of the software requirements and therefore can make sure the task was even correctly understood in complex domains.
Developers might be good at writing code, but they can never be unbiased to whether they understood the acceptance criteria correctly.
And does it do something it is not supposed to
Please add subtitles
There are already subtitles (captions) in most YT viewers you should see a "CC" icon, if you click that the captions should work.
The ads that run forever are ridiculous.
Not everyone has direct access to hit the skip button very easily all the time
Slightly disagree on part where you chuck manual testing to QA engineer, where as they can delegate this to other team members and work on automated testing.
Overall an amazing insight, probably best explanation of shift left. I’m going to share this with my team.
I don't think I said "chuck manual testing to QA" 🤣
@@ContinuousDelivery at 12:49 : 3 amigo meeting. A little sleight of hand 😉
I definitely think Dave has had some unpleasant experiences in the past with testers finding bugs in his code.
More like the other way around, unpleasant experiences of testers NOT finding bugs. I want to find lots of bugs , then I can fix ‘em! 😁😎
@@ContinuousDelivery On a human level, I don't think anybody really likes someone finding fault in something they have created. I think this is the main reason why the software testing industry is in such a mess/quagmire of bs. Love it or hate it though, fault finding is a necessary cog in the corporate money making machine.
This only works if you have a single isolated project. I work in bank where your component is part of a larger system. You don't know if everything works until your code is running in a testing environment. And it's likely that you need to integrate your component with other components in development. So for us, this idea of continuous testing doesn't work. You have multiple teams working on separate components that need to be integrated. So you need an agreed QA environment and QA testers testing once the changes have been deployed into QA.
I'm pretty sure this is true of most complex applications like games. I'm pretty sure that the devs of Starfield change components then create QA builds or release candidates and test those with their QA team.
Not the case I am afraid. I built one of the world's highest performance financial exchanges, I have worked with several banks that work this way. This is how lots of big orgs, building complex software work, it is a matter of how you organise development, and how you design the software.
A few more name-drops: Google, Facebook, Netflix, Amazon, Spotify, Tesla, SpaceX, CitiBank... It's a long list!
The video is very compelling, but shows a very shallow understanding of testing.
Testing is akin to design (not manufacturing). How do you prevent problems in abstract ideas like design? You can carry out *thought* exercises. However, it is much more powerful to experience the implementation to find problems.
You cannot automate that process.
The idea of QA being a gatekeeper was debunked in the 1980's by testers. Promoting that idea 40 years later is negligent to the extreme.
In the whole video there is no acknowledgement of the role of the individual in testing - how do you think about risk, how do you come up with creative ideas about failure? Have any development thought leaders contributed to that process?
Lastly developers like coding. They are not motivated to spend hours to learn software in order to find problems.
Good idea to read Kaner, Weinberg, Satisfice, Developsense
Did we watch the same video? He is actually suggesting that QA should NOT be gatekeepers and instead should be involved in the whole process of requirements and testing, along with developers and product owners.
@@cristi724 the point is he should not be talking about that as if it is something profound. This is a common topic when developers, agilists, devops talk about 'qa'. No good tester/QA is a gatekeeper. As I said, that idea was debunked around 40 years ago. Suppose someone decided to talk about how Devops is changing development. Then they talk about the importance of naming variables and using source code control.
Also when you talk about non-issues, it means that you don't talk about what is really important. Is there *any* understanding of testing/QA?
Shallow understanding comes from not evolving from a legacy practice to an agile world.
I disagree a little here. I think that there are different personal skills on a good dev team, and that both the dev and QA work should be shared without labeling one person as QA. Everyone on a dev team should be able to write code. Everyone has somewhat different talents within that specification.
But boy do I agree about the importance and purpose of manual testing! I was at Microsoft when they cut all of their FTE manual testers. What a mistake that was! Pure ignorance on billg’s part.
Wasn't the "Three Amigos" approach (Gojko Adzic) trendy 10+ years ago? And do we need to still keep insisting on specialization/roles nowadays (e.g. dedicated QA role)? I have seen enough issues with balancing highly specialized team members inside a team, or having people fill their time with manual tests ("exploratory"), or writing their own semi-automated test suites with pretty bad code quality to still believe in such a specialization ... Dev-only teams (+ removal of many other fluffy roles like Scrum Masters, Product/Project Managers/Owners, Sysadmins in the self-service cloud age, etc.), with BDD/Gherkin/full test automation and end-to-end responsibility is the way to go IMHO ...
First