How do you handle state with this approach? Like, getting the orders depends on the state of the database, i.e. on the database updates that were done previously. Without state, this looks like it's good for type checking only, and I prefer static checks for that.
This is a great question, and is what "Provider States" are designed for. They allow you to setup your system to verify certain API scenarios, without requiring you to coordinate tests to run in a particular order. See docs.pact.io/getting_started/provider_states and our example projects for more.
You mentioned you don't need a dedicated test environment, how would you cater for a hybrid scenario where you want to test an API in a monolith application that also calls a microservice from the provider pov?
does the example mentioned mean that it will test against the provider with order id = 10 ? what it the provider (its SOR actually) doesnt contain and order with this ID? or what if it returns data with values other than the values stored in the repository contract ? is there an option to set these values dynamically so that the values while mock testing the consumer is different than these while testing the real provider ?
Yes, there are two Pact ways to handle this - the use of Provider States and the use of generators. This is one of the neat ways Pact works around the "data fixture" issue. See docs.pact.io/getting_started/terminology#provider-state, docs.pact.io/provider/using_provider_states_effectively and example of dynamic values from the provider here: pactflow.io/blog/injecting-values-from-provider-states/
Sometimes these terms on testing gets confusing. So what is a unit test to you? I consider unit tests "testing the smallest piece of logic within a class or module ". Whereas a component test is "testing the smallest deployable code possible. A component test should run in its own jvm or container". Also does pact test actual live endpoints of an api or just mocked?
Yes, testing terms are confusing :). Strictly speaking, a Pact test on the consumer is closer to an integration test (middle of the pyramid) because we expect the test to also make an http call and crosses a boundary, but it's more helpful to think of it as a unit test in terms of writing a unit test of your code, and allowing it to make the HTTP call. We usually draw in on the boundary of "unit" and "integration" on the classic testing pyramid. On the provider side, it's closer to a component test in your definition. A consumer test only uses mocks of the provider, which produces a contract. The contract is then verified against the real provider (usually run locally, not in an integrated environment). Hope this helps clarify things!
1. We still need to deploy the actual Provider to run provider tests. right ? so why we say it is equivalent to unit? 2. What we do to the Provider dependencies ?
No, the provider should be tested in a unit test-like context and not in a deployed environment. It should be able to run on your development machine - this is how you get the fast, reliable feedback. Deploying to an environment makes it more of an end-to-end / functional test, which we are trying to avoid with contract testing. This means any dependencies should be stubbed out, and they should then have their own set of Pact tests to ensure they are properly covered also. See docs.pact.io/faq#how-do-i-deal-with-a-situation-where-there-are-multiple-systems-involved-in-a-scenario for more on that.
Hi I am having issues with understanding what to define in the providerstates when creating a interaction via consumer. So it is all good when the domain model in frontend almost has a 1-1 mapping with the backend but it is difficult to understand once the frontend requets a data object that is not pure in the backend but rather build but different sub resources that the frontend (should not and do not want to care about). How should you define providerstates in that case?
Think of provider states as a precondition for the consumer test to pass (e.g. "user is authenticated" or "user has a positive credit balance"). The states shouldn't have implementation details in them, it is up to the provider to determine how to implement a provider state handler. The provider could seed a database (if it's a 1:1 mapping) or stub a 3rd party API if it's further downstream.
Where do we read the contract from? can we read it off proto files?
3 роки тому
On the first slides it says that a REST/JSON/XML is expected to be checked against. By proto file I assume you mean gRPC which I thinks it’s not supported by Pact. Because it’s a binary response.
We're currently building support for gRPC (via our plugins proposal), so it's coming soon: See also: * github.com/pactflow/roadmap/projects/1 * github.com/pact-foundation/pact-specification/issues/83 * pact.canny.io/feature-requests/p/support-grpc
Contract testing is more than just ensuring breaking changes. The tests themselves are like any other good test - they apply stress to the design and structure of your code. If Pact tests are hard to write, that is a sign that your code is not very testable and modular and should be improved. There are ways to generate tests though, but you should only consider this once you know what you're doing. You may also like our bi-directional contract testing feature which simplifies this workflow / allows you to re-use mocks and existing testing tools: docs.pactflow.io/docs/bi-directional-contract-testing
I'm a bit confused with the example given from the provider side. Is the Pact verifier checking against the provider or a mocked response from the provider? If so, surely your argument about the other dependant services falls a bit short as the provider is still potentially running against those other services? I can see the value in the consumer side though
Actually I think its nonsense what I've put there.. as at the time this is tested you'd have the consumer up and running as part of the build pipeline right?
It is recommended to stub out external systems as part of contract testing (see also docs.pact.io/5-minute-getting-started-guide#scope-of-a-provider-pact-test and docs.pact.io/provider#stub-calls-to-downstream-systems). This increases the reliability of those tests and makes them more unit test-like. Of course, if they are other APIs, you should have a separate set of contract tests to cover those, and so on until you reach a leaf node. -Matt
I'm not sure I understood the second question, but we'd be happy to chat in more detail over at slack.pact.io and clarify any questions you have. P.S. you have the voice of an angel 🎶
In this case, yes that's correct. For clarity, it's _not_ a stub, it's a mock. A mock is a validated stub. That is, if your code does not call the mock the test fails. This is important, to ensure we're capturing a valid contract.
@@PactFlow thanks for the clarification, i'll read on more to learn about it. I was lookin into it for my company's QA team to use along with Cypress, but it seems this is more suited for a Dev team to use on top of their Unit testing., is that a correct assumption? thanks
@@ricp Yes that's right. Pact is really designed to be used in a unit test setting (i.e. white-box style testing), when the code is being authored. Pactflow has some features that could be authored external to the code base in a more black-box style test.
I couldn't finish that video, but the first part it's just nonsense. You are saying to drop integration and e2e tests and do just unit tests (you call it contract, but basically it's just unit tests because use mocking everything). Unit tests go along side integration and e2e, and would never replace it. It's just like you created a new quality control tool to verify car parts quality and you're saying: "Our tool is so good, you don't need assemble car's anymore and test it. We can just sell it, because all parts are tested perfectly already".
Hi Abbey, that's certainly not the intent. Video 1 intends to show the problems with e2e tests, and video 2 is about showing how contract testing works as a mechanism. It's true that contract-testing with Pact uses unit tests on either side of an integration point, they are better than unit tests in terms of guarantees, but less than e2e tests. Contract testing does not cover functional testing, which is another key benefit of e2e tests. Instead of using e2e, we recommend using a combination of tests (functional, contract etc.) and reducing if not removing e2e altogether. See Video 4 where we talk about a more holistic view of a testing strategy. For clarity: contract testing doesn't entirely replace end-to-end tests alone. Hope this helps! - Matt
![[Introduction to contract testing - Part 3] Demo - Contract testing with Pact JS](http://i.ytimg.com/vi/6Qd-kq1AzZI/mqdefault.jpg)
![[Introduction to contract testing - Part 3] Demo - Contract testing with Pact JS](/img/tr.png)
![[Introduction to contract testing - Part 1] The problem with end-to-end integrated tests](http://i.ytimg.com/vi/U05q0zJsKsU/mqdefault.jpg)
How do you handle state with this approach?
Like, getting the orders depends on the state of the database, i.e. on the database updates that were done previously.
Without state, this looks like it's good for type checking only, and I prefer static checks for that.
This is a great question, and is what "Provider States" are designed for. They allow you to setup your system to verify certain API scenarios, without requiring you to coordinate tests to run in a particular order. See docs.pact.io/getting_started/provider_states and our example projects for more.
You mentioned you don't need a dedicated test environment, how would you cater for a hybrid scenario where you want to test an API in a monolith application that also calls a microservice from the provider pov?
does the example mentioned mean that it will test against the provider with order id = 10 ? what it the provider (its SOR actually) doesnt contain and order with this ID? or what if it returns data with values other than the values stored in the repository contract ? is there an option to set these values dynamically so that the values while mock testing the consumer is different than these while testing the real provider ?
Yes, there are two Pact ways to handle this - the use of Provider States and the use of generators. This is one of the neat ways Pact works around the "data fixture" issue.
See docs.pact.io/getting_started/terminology#provider-state, docs.pact.io/provider/using_provider_states_effectively and example of dynamic values from the provider here: pactflow.io/blog/injecting-values-from-provider-states/
Sometimes these terms on testing gets confusing. So what is a unit test to you? I consider unit tests "testing the smallest piece of logic within a class or module ". Whereas a component test is "testing the smallest deployable code possible. A component test should run in its own jvm or container".
Also does pact test actual live endpoints of an api or just mocked?
Yes, testing terms are confusing :). Strictly speaking, a Pact test on the consumer is closer to an integration test (middle of the pyramid) because we expect the test to also make an http call and crosses a boundary, but it's more helpful to think of it as a unit test in terms of writing a unit test of your code, and allowing it to make the HTTP call. We usually draw in on the boundary of "unit" and "integration" on the classic testing pyramid. On the provider side, it's closer to a component test in your definition.
A consumer test only uses mocks of the provider, which produces a contract. The contract is then verified against the real provider (usually run locally, not in an integrated environment).
Hope this helps clarify things!
1. We still need to deploy the actual Provider to run provider tests. right ? so why we say it is equivalent to unit?
2. What we do to the Provider dependencies ?
No, the provider should be tested in a unit test-like context and not in a deployed environment. It should be able to run on your development machine - this is how you get the fast, reliable feedback. Deploying to an environment makes it more of an end-to-end / functional test, which we are trying to avoid with contract testing. This means any dependencies should be stubbed out, and they should then have their own set of Pact tests to ensure they are properly covered also. See docs.pact.io/faq#how-do-i-deal-with-a-situation-where-there-are-multiple-systems-involved-in-a-scenario for more on that.
and can it be used to test non functional and repeatitve tests ... like testing the consumer quota ... CORS options, etc ...
No, that's not what contract testing is about. See also docs.pact.io/consumer/contract_tests_not_functional_tests
Hi I am having issues with understanding what to define in the providerstates when creating a interaction via consumer. So it is all good when the domain model in frontend almost has a 1-1 mapping with the backend but it is difficult to understand once the frontend requets a data object that is not pure in the backend but rather build but different sub resources that the frontend (should not and do not want to care about). How should you define providerstates in that case?
Think of provider states as a precondition for the consumer test to pass (e.g. "user is authenticated" or "user has a positive credit balance"). The states shouldn't have implementation details in them, it is up to the provider to determine how to implement a provider state handler. The provider could seed a database (if it's a 1:1 mapping) or stub a 3rd party API if it's further downstream.
Where do we read the contract from? can we read it off proto files?
On the first slides it says that a REST/JSON/XML is expected to be checked against. By proto file I assume you mean gRPC which I thinks it’s not supported by Pact. Because it’s a binary response.
We're currently building support for gRPC (via our plugins proposal), so it's coming soon:
See also:
* github.com/pactflow/roadmap/projects/1
* github.com/pact-foundation/pact-specification/issues/83
* pact.canny.io/feature-requests/p/support-grpc
and is there any way to expedite writing the contract (i.e. test cases) ... because I believe it is a bit verbose
Contract testing is more than just ensuring breaking changes. The tests themselves are like any other good test - they apply stress to the design and structure of your code. If Pact tests are hard to write, that is a sign that your code is not very testable and modular and should be improved.
There are ways to generate tests though, but you should only consider this once you know what you're doing.
You may also like our bi-directional contract testing feature which simplifies this workflow / allows you to re-use mocks and existing testing tools: docs.pactflow.io/docs/bi-directional-contract-testing
I'm a bit confused with the example given from the provider side.
Is the Pact verifier checking against the provider or a mocked response from the provider? If so, surely your argument about the other dependant services falls a bit short as the provider is still potentially running against those other services? I can see the value in the consumer side though
Actually I think its nonsense what I've put there.. as at the time this is tested you'd have the consumer up and running as part of the build pipeline right?
It is recommended to stub out external systems as part of contract testing (see also docs.pact.io/5-minute-getting-started-guide#scope-of-a-provider-pact-test and docs.pact.io/provider#stub-calls-to-downstream-systems). This increases the reliability of those tests and makes them more unit test-like. Of course, if they are other APIs, you should have a separate set of contract tests to cover those, and so on until you reach a leaf node. -Matt
I'm not sure I understood the second question, but we'd be happy to chat in more detail over at slack.pact.io and clarify any questions you have.
P.S. you have the voice of an angel 🎶
so, on 6:22 are those Mocks responses mainly stubbed http responses?
In this case, yes that's correct. For clarity, it's _not_ a stub, it's a mock. A mock is a validated stub. That is, if your code does not call the mock the test fails. This is important, to ensure we're capturing a valid contract.
@@PactFlow thanks for the clarification, i'll read on more to learn about it. I was lookin into it for my company's QA team to use along with Cypress, but it seems this is more suited for a Dev team to use on top of their Unit testing., is that a correct assumption? thanks
@@ricp Yes that's right. Pact is really designed to be used in a unit test setting (i.e. white-box style testing), when the code is being authored. Pactflow has some features that could be authored external to the code base in a more black-box style test.
How Pact knows what is the correct provider response to consumer request?
The contract is authored by a human (it's a unit test) and is verified later on by the provider. Check out our introductory videos for how it works.
What tool is used to draw these diagrams?? Really good
Google slides ;)
What's that annoying dot? 🤔
It's the mouse 🖱️
🤣 🤣
🤣
Well its just a contract testing we can place it in the testing pyramid between unit tests and integration tests and move one - geez
I'm sorry, is there a point you're trying to make? The video is to help people understand what contract testing is and how it works.
I couldn't finish that video, but the first part it's just nonsense. You are saying to drop integration and e2e tests and do just unit tests (you call it contract, but basically it's just unit tests because use mocking everything). Unit tests go along side integration and e2e, and would never replace it. It's just like you created a new quality control tool to verify car parts quality and you're saying: "Our tool is so good, you don't need assemble car's anymore and test it. We can just sell it, because all parts are tested perfectly already".
Hi Abbey, that's certainly not the intent. Video 1 intends to show the problems with e2e tests, and video 2 is about showing how contract testing works as a mechanism. It's true that contract-testing with Pact uses unit tests on either side of an integration point, they are better than unit tests in terms of guarantees, but less than e2e tests. Contract testing does not cover functional testing, which is another key benefit of e2e tests. Instead of using e2e, we recommend using a combination of tests (functional, contract etc.) and reducing if not removing e2e altogether. See Video 4 where we talk about a more holistic view of a testing strategy. For clarity: contract testing doesn't entirely replace end-to-end tests alone. Hope this helps!
- Matt
thanks, i have same question