The cleanest way to use Docker for testing in .NET
Вставка
- Опубліковано 24 лип 2022
- Check out my courses: dometrain.com
Become a Patreon and get source code access: / nickchapsas
Hello everybody I'm Nick and in this video I will show you how you can quicly spin up and tear down docker containers specific to your test executions. This is only a fraction of what I show in my Integration Testing course so if you want to learn even more cool topics like this check nickchapsas.com out.
Workshops
NDC Oslo | 26 - 30 Sept | bit.ly/ndcoslo2022workshop
NDC Sydney | 10 - 14 Oct | bit.ly/ndcsydney2022workshop
dotnetdays | 20 - 22 Oct | bit.ly/dotnetdays2022workshop
NDC Minnesota | 15 - 18 Nov | bit.ly/ndcminnesota2022workshop
NDC London | 23-27 January 2023 | bit.ly/ndclondon2023
Give TestContainers a star on GitHub: github.com/testcontainers/tes...
Don't forget to comment, like and subscribe :)
Social Media:
Follow me on GitHub: bit.ly/ChapsasGitHub
Follow me on Twitter: bit.ly/ChapsasTwitter
Connect on LinkedIn: bit.ly/ChapsasLinkedIn
Keep coding merch: keepcoding.shop
#csharp #dotnet #testing
Amazing introduction. Thanks Nick!!
I use this package for a long time and it saves me a lot of effort configuring containers for testing different pars of my systems.
Here is a small helper method to get the next free port with a significantly lower risk to choose an already used port :
public static int GetNextPort()
{
// Let the OS assign the next available port. Unless we cycle through all ports
// on a test run, the OS will always increment the port number when making these calls.
// This prevents races in parallel test runs where a test is already bound to
// a given port, and a new test is able to bind to the same port due to port
// reuse being enabled by default by the OS.
using var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Bind(new IPEndPoint(IPAddress.Loopback, 0));
return ((IPEndPoint)socket.LocalEndPoint!).Port;
}
you don't need custom code, you can use this overload (on the TestcontainersBuilder) and it'll choose a random port for you:
.WithPortBinding(1433, assignRandomHostPort: true)
Excellent content, thanks for sharing to us.
That is absolutely excellent. Thanks Nick. Definitely going to share this with the rest of the team at work. Another clean and clear video.
Super slick! Love it, thank you.
Excellent, excellent video thank you very much Nick!
Amazing content Nick! Thank you!
This is a brilliant video. Thank for the info
Awesome content! Thank you for sharing 👍
Thanks for helping so much with this video in simplifying testing in my job, this seems like a great approach!
Heh, what a coincidence. I just discovered TestContainers a week ago and started using it in our integration tests. It's pretty darn awesome. Great content as usual, Nick!
Hey! I've been using Fluent Docker(or Ductus for some) for running integration tests inside a Docker container for over a year now and it has been great so far. I didn't even know there are any alternatives to it before I stumbled upon on your video. However, after watching your video, I would love if you could do a small parallel between the two of them as I am sure there are some differences or other things that one does better over the other. Great video again and can't wait to see you in Romania!
Great video Nick! For our integration tests we need a good amount of static and transactional data setup. We settled on a similar strategy to yourself where we clean DB's per test collection. However, we still found creating new DB's too slow, so we instead settled on using Respawn to intelligently clear transactional data while leaving static data intact. This has worked great for us. One disadvantage though, is that we can't parallelise the tests as you've shown.
Respawn is getting a video next month. it’s pretty cool
I will have to test Respawn. I created my own solution. It uses database snapshots and the database is "seeded" from a .SQL script file. The first test create the database and make the snapshot; other tests revert back to the snapshot before running.
Why not just create a local image with static data already there, and run containers based on that image? I haven't tested this though
@@mohammadazhdari4469 It's a perfectly fine solution it just needs a bit more handholding because you have to keep it up do date
I have just finished implementing a similar approach, but to handle parallelism, we are using 1 db container instance per test assembly, then 1 Sql database per testFixture. We then override the connection string in the OneTimeSetUp of each TestFixture. It allows parallelism at the class level while avoiding too much RAM usage. ultimately I'll want to use only 1 container for all the assemblies, but it requires inter-process synchronization (Mutex) for making sure we create the container only once.
We have also appended part of the hash from the list of migrations to the name of the container, so whenever the list of migrations is modified, a new container is bootstrapped.
Another thing I noticed is it's best to leave the container running between runs (at least during local development), because otherwise it takes at least ~5 seconds to start on each run.
One thing I found lacking with the TestContainers is the lack of support for already-running containers. Because of that, we had to implement some of the code manually and use a wrapper class to handle both cases.
This issue explains the problem: github.com/testcontainers/testcontainers-dotnet/issues/506
Thank you! looks really convinient and easy!
I realy real exited to use this superb library. It is exactly things that I wanted.
Would be nice to have a video on the latest version of Testcontainers as it makes this video obsolete :) Thanks for your great work
This is gold. Really. No more need to complicate with multiple DB services in CICD or docker-compose just to bring up databases. Thanks!
Nick is great at showing us hidden gems like this. Great work!
Hi Nick! We have been using TestContainers in our team for six month. Awesome library to facilitate and standardize using of docker containers in NET. The same running procedure for integration tests from both IDE and CI/CD cannot be overstated:) Also I want to mention that we are not using "one class - one container" approcah, because most of the time muiltiple tests should rely on the same data in database. We ended up with custom abstactions for container and dbcontext that give us ability to create one or several databeses per any scope of tests. We have separate nuget packages for EF6 and EFCore due to different approach for configuring and providing DbContext instances in integration tests. Code base is super small but very useful.
Do you use the containers inside a CI/CD pipeline such as Azure DevOps? If so, how?
It would be good if you can write a post about how you do it ( custom abstactions for container and dbcontext that give us ability to create one or several databeses per any scope of tests)
Amazing video, thanks
Every time I watch one of your videos it reminds me of how much of a cowboy developer I am! 😀 I really should sign up for some of your courses.
great video, please make more content about docker.❤️
Semantic test case naming convention. A good one 👍
thanks nick this helped.
Nick, I have a request for your videos, please turn on tracking of the selected file in the solution explorer pane. This would help to better keep track of the selected file when you sometimes go a bit fast between file.
Love your content.
See you at NDC Oslo!
thanks for the video
This looks similar to FluentDocker, that you presented in one of your videos "5 open source .NET projects that deserve more attention"
Very useful, Thank you
Great video Nick as always, one question please, how did you run the initial migration scripts, I see there is DatabaseInitializer but I can't see where you used it
Great video Nick. Don't forget peeps .. RavenDb is another awesome (and so underrated) Document DB which could also be used here. Nice :)
There are probably 4-5 alternatives that I’d use before I even considered RavenDb for any real world application
@@nickchapsas Eeks - that's pretty .. damning :(
@@nickchapsas Hey Nick. Clearly you are an advocate for C# (me too). I have yet to see a database client that comes even close to RavenDB's C# integration. My words truly cannot give enough justice to RavenDB. As a fellow C# developer, I highly recommend you to give it another try.
Not to mention the fact that I successfully launched a video game in production using RavenDB. Had no problems handling 17k+ concurrent users. I picked RavenDB for a good reason. The C# integration was above and beyond. Try the MongoDB client for 10 minutes and you will see how terrible it is.
@@nickchapsas could you name those alternatives? have you tried ravendb yet?
So i have tried this approach before. The problem i have with it is that you end up running several database instances on you local machine which can get slow when the number of tests increases. Postgres has a habbit of restarting itself one time before it comes online if you mount scripts to add extensions like plv8, which means you need to wait 10 seconds before you can run the tests. Its also problematic if you are dependent on other services which need to built from dockerfiles in the same project. Maybe you need to for example run dbmigrations before you can run the tests.
The nice thing with docker compose is that it will fire up all your services, you message queue, you amazon stack, you databases, your apis, everything with one command and then you can just run your tests. Of course, some of them cannot run in paralel. But often you can work around it by for example testing different scenarios on different users or different accounts. So even if each scenario is sequential you can run several scenarios in paralell.
Nice video and thanks for introducing this nuget package to us. But in version 3.x.x code for setting TestContainers has little changed now there are separated nuget packages base on what you want to have in container. For example in video Testcontainers.PostgreSql need to be installed not just Testcontainers.
This was a brilliant video where you could present a great tool to improve productivity on programming and running integration tests, every developer suffer when setting up test env to do that.
I would like to leave a suggesting: this kind of test is built in the way where might increase the consumption of resources, instantiating new containers every time a new test method is run. Wouldn’t it be more performant if the customerApiFactory class was instantiated once, and then when all customer’s tests was done, the factory class was killed?! Yes, I know that, in this case, the test implementation should implement some kind of cleanup cleanup!
Wouldn’t have another approach to consume less resources?
Great stuff as always. Quick question - How did you create the table in the database?
Assuming you're using EF Core and not Linq2Db, for example, you can use the DbContext to get the migrations and migrate if any migration is pending. Otherwise you can call EnsureCreated which will only work on a fresh database with no tables.
A nice way to use TestContainers in integration testing is in conjunction with xUnit collection fixtures.
Can you say a bit on why it is so? What makes use of xUnit "nice"? :-?
@@RajaKajiev not XUnit, but Xunit Collections
@@PeriMCS ok, meditate on this I will (c) Yoda
Hi Nick, thanks for your video. What do you do with db mirgations when using Testcontainers? Are they executed every time the test suite is run, or is there a way to reuse a container with migrations applied once?
It depends on your execution model and whether you wanna run all the tests in parallel or in sequence
Nice video, but can you use timestamps next time? It's a very handy feature that helps people finding the parts they do know / didn't know yet, saving valueable time
Great video! But i got somewhat related question. Whole weekend i was trying to implement transactions for my integration tests which are using in memory client provided by WebApplicationFactory. I tried to wrap my test code in TransactionScope but nothing work. I also tried to re-register my dbcontext as singleton for my integration tests webapi instance so i could retrieve transaction and rollback it but it also wasnt a solution. I ended up deleting and recreating db after each integration test. Do you guys have any snippet that implements transactions for integrity testing using WebApplicationFactory?
How do you create the database and all the tables?
Do we run all migrations in our project?
Great video, thanks!
As I understand you generate your database schema from C# code. What would I need to do if my database schema was generated by a set of SQL commands or some external tool? Is it possible to apply "external" database migration?
Sure thing, you can either bake that on the docker startup level. Simply call the scripts towards the database your gonna run the tests against. You might wanna look at a library called Respawn to allow you to get back to a given state for every test
One question which came in my mind regarding those containers is: What if I want to have the testdatabase created by an ef core migrator?
I maybe overheard it, but how do you approach db deployment to your database in test container? we use Database project in our company and that'd need to be deployed first there.
Great video Nick. How would you do it if some tests do need to run sequentially and need the information from the results from other tests?
Standard practice is to keep tests independent of each other
Great content as always Nick. As recently as September 2021, you highlighted FluentDocker for this same niche (that of integration test Docker orchestrators). Do you now favor TestContainers over FluentDocker, or do you see a use for both of them?
It depends on the usecase. I use both and in my integration testing course I show both usecases
I'm working on existing application which uses lots of stored procedures and few more tables. I was able to automate the recreation of these sql objects upon starting of integration tests using EF migration that fires this sql scripts on `Migrate`. If I do this for each invidiual tests, this might add a lot of time provisioning each instance of docker db server instead of re-using the same instance and just do a cleanup for each test. The side effect on our approach as you mentioned is we can't run it on parallel unless we guaranteed our test data like GUID idenfier id test data. Even with this approach, our integration test completion time is still fast. I might do some comparison between these approach running on parallel and see performance gain. Also not sure how much cpu and memory utilization we gonna consume on the CI pipeline, let says if we spawn 10 instance or whatever the parallel number the test is running.
You could use .EnsureCreated() instead of applying migrations. It's way faster
Yo, if you have some test objects where you make tests with, how do you manage when you try to insert same object or when you already inserted an object and run the test again?
Great! How can you start a db already populated?
Thanks for the yet another great video. Is there a way we can run these in the devops pipelines? What are the pre-req for the same?
The pipeline needs to support Docker but that's it. I've used it with TeamCity without any problem
This makes me want to write an integration test. Thanks
I like this, interesting
Hello Nick, very nice video, thanks! One question though, since it is creating a database container for every test, it could get out of hand easily when you have lots of tests. Is there an option to counter this?
In this example it doesn’t create a new database for each test but rather one database for each test set which in this case it is a class. If you group your tests logically and do minimal clean up, you can run less databases like I show in the example. You can also run one and clean up after every test assuming you configured the tests to run one after the other. You control this fully and it is a balance between how much you wanna cleanup and how much you can parallelise
@@nickchapsas Yea, sorry, it is creating one database for each test set, not for every test, my bad but still it can be a lot when you have more and complicated tests. So, is there any option in the package to differentiate and control the created containers? For example, putting an attribute to use a naming instance ([ContainerName="Container1"] for example) over the tests to use an existing one or create a new one. Or should we do this manually?
To answer my own question, after couple of experiments, I think best way is to handle multiple setups and prevent creating new databases for every setup is to use the solution @Daniel Mackay gave (using Respawn).
Great tutorial as always, i just wanted to ask if all these scenario are part of your current courses; if no then will it be updated with these new features?
Yes my integration testing course covers this in way more detail and it shown another way to do this for applications with a front end
@@nickchapsas now i just need to save to get the bundle course
We currently use docker compose for our tests to fire up a mysql instance and then create a new db for each suite of tests. I could be missing it, but I would think creating one container with multiple DBs will be more performant than bringing up/down containers?
You won't really see a difference because of how memory is allocated on containers. It is the database itself which is the slow part, not the spinup of containers
Hi Nick, we do have similar problem, however we are using UI testing and cant run the tests in parallel. Any solution for these scenario, great if you could do a video on running UI testing parallel.
Great content. You forgot the link to the nuget package in the description.
Sorry for that, it’s there now
Great content.
How can such test suite be running in a CI/CD?
Is there ability to deploy the tests as docker image and run it via cmd line without having dotnet test runner environment?
Absolutely. I’ve been running this in my CI pipeline in TeamCity
Interesting! How did you run the initial migration scripts (ie creating the customer table)?
I'm guessing it's somewhere in the DatabaseInitializer.cs file in the Database folder of the project.
You can also have it be in the ConfigureServices in the webhost builder in the WebApplicationFactory class
Can you do something like having a test run an having to build the config (app settings) every time we run a test?
I tried reproduce the steps and an exception was thrown (Testcontainer has not been created), the database used is MySql 8
I noticed the method InitializeAsync of IAsyncLifetime isn't being reached, is there some chance of the issue being related at MySqlTestcontainer?
Great video! Could this be combined with entity framework?
We have a code first setup and this would be great if the docker database could always be the same as the real database
Sure it can
Hey Nick, how would you run these tests in CI/CD such as Azure Devops pipelines? Is it possible to use the cloud build agent as a Docker host?
I've been using TeamCity for that, and in TeamCity it is possible to have docker running in the agent yes
@@nickchapsas Are you running the tests in the host? We have a problem where we run the tests while building the docker image (so in docker build there is a command that runs the tests) which makes it really hard to use this tool. Have you faced the same problem?
Very cool tooling.
A) where does the schema load? And how does that affect test time?
B) in Java we can specify to keep the container live after the test is complete. Is there an option like that?
C) in Java the default flow uses transactions to keep the DB "clean" between tests. Is that an option here?
A) in this case migrations run on startup and create the schema. They are usually very fast. If you have a very lenghty migration time, simply create a docker image with the schema and all the data pre-seeded and use that for testing.
B) Yes, if you don't dispose the container in the dispose method it will stay up
C) In C# there are multiple ways to interface with the DB so you control the transaction scope yourself. This example shows how you can create a clean DB every time so you don't have to worry about cleaning data.
B) .WithCleanup(false)
Can we run tests with azure devops pipelines ?
Yup, this is pretty much what we do only we use NUnit to set up one db for everything. Having a db per class would be… a lot of db’s 🙃
Thx a lot, I briefly look through Azure DevOps docs and I can't realize is this approach could be used with AzureDevOps
If your Agent has docker up and running, it will run it without problem on Azure Devops.
Nice video, thanks ! The only problem I see with this package is that you can't run the tests themselves in docker right ?
If you need to control docker from inside of a docker container, you can bind mount docker control socket into the container.
Testcontainers is cool.
Was watching a vid by Jason Taylor where he used respawn library that intelligently wipe your tables after Ur test finish running but this means u need to have two database active at time. Guess his way and ur way both works
Respawn is great for this I’m planning to make a dedicated video on the library
How about the performance? Isn't it too heavy to instantiate multiple postgres dockers for each unit test.
These are not unit tests, these are integration tests. It depends on the metal you are running on but with the ram we have today, you can run a lot of them before you have any problem especially with them being created and deleted as the tests are running
Instead of running a new container each time you need to execute an integration test (or a set), what about using the same container e and creating/deleting a new db instance with a different guid name?
The container itself isn't the "heavy" thing. Its the data that you gonna need to seed into it that might take time. DB Instances are effectively as cheap as containers or at least you wouldn't see a visible difference.
how could I run this in a azure pipeline that uses service frabric? it is possible....this is a great tool I'd like to bring to my team
What if I have lots of migrations to run to create a new db? Is it still a good way to go?
Sure thing. You can deal with it in multiple ways from a docker image that has them already run, to backup restore on startup to using a library like Respawn to go back to a snapshot
I assume your WEB API creates a DB schema. If so, how do you synchronize it in a microservice production environment?
You can run your migrations and seed data on startup or have a docker image with all the schema and data pre-seeded
@@nickchapsas thank you for the answer! I very much appreciate that.
We tried migrating DB in a microservice itself, but when we scaled it to multiple instances, we had to lock migration somehow. And this adds unnecessary complexity.
Also, we tried creating custom images, but this didn't work too. Because a script that creates schema starts at the same time as a DB server, so you can't wait for an open port, you should wait for the last migration applied, which is hard to determine.
I'll research this lib, maybe it can help with my problem.
How does this compare to Respawn, is it faster or slower? With Respawn you can't have parallelism but you're not creating/destroying DB for each test. This looks awesome but it seems like it would be resource-intensive and slow since it has to create DB, apply migrations, run a test, then destroy DB - all this for each test.
It's not for each test. It's for each test collection
That's the problem with XUnit, there is no concept for test session setup like NUnit has. XUnit closest similar feature is collection fixture which defined by code statically.
Awesome one
Can you please attach the github repo code sample to use it
The code is available to my patreons
What happened to the tests approach
- seeds test data (arrange)
- performs a test scenario (act)
- checks the data (assert)
- deletes test data used in this scenario after execution (teardown, clean up) ?
When test data not spreaded in multiple scenarios - everything works fine. Each test works with it's own test data. Or I missed something?
I think I got the idea - with this approach you don't need to think about managing test data in DB at all
By curiosity : why don't you just replace service instead of remove and add ?
Removing based on typeof will remove all the implementations. You'll usually just have one but there are cases where you might have more than one
btw nice video. lookin forward to swtich from SQLite inmemory to this and give it a try
.
Hi, Nobody asks how much RAM these tests consume and if the container go down, i suppose the test will fail.
The container won't go down unless the test executes and the RAM is relative to your docker configuraiton
Kubernetes tutorial please
And now we need to talk about memory for 10000 tests =)
Hey Nick, the random ports looks like a really bad practice. It can be a reason for flaky tests, which are unreliable and can mask real problems
I think that writing custom port error handling might be the only solution
Which random ports? The ones that the library picks of you automatically? Those internally are checked for availability so it will never pick ports that are not available.
@@nickchapsas Ok then, thats cool.. nice video, thanks
Seems like the video content is outdated because of test containers updates
Who you are teaching Docker with vscode?? Be serious pls
Be serious, where do you see vscode in the video?
First of all , he didnt teach docker at all. He presented a nuget package called "TestContainers" to simplify your integration tests. Second, there is no vscode, its called "Rider" from jetbrains. Finally one question, what is your point? Using vscode is not bad habbit. Use the IDEl, you feel comfortable with.
Or save yourself a lot of headache and get rid of all this 1970s "SQL" nonsense.
No-SQL databases are far superior and much more easy to integration test against.
You clearly not now why RDBMS are used and when to go to NoSQL. For you it's just SQL and NoSQL.