This is great! Would be great to see a similar video on how to go about understanding the limits(load test) of an event/message based application, if possible
What would be interesting is if you included an example of 'here is an example of an API that where stressing it caused it to fail.' Then apply some fix, run it again, and compare resulting metrics. But great video/idea.
This video is a setup for a future video in which I will use a soak test to identify a memory leak in .NET. Just needed to have this one out first for reference.
I'd love to see load testing integrated on a website. I am building more APIs than I used to but I still manage a good number of websites that don't consume a dedicated API.
We use NBomber. It is C# based (well, there is F# underneath). As I understand, it should have very similar functionality. But it is C#. We were very reluctant to support anything in not-C# and NBomber covers most of our needs. There was a bit of a struggle with implementing load tests. But in 9 cases out of 10 it was because of a lack of performance testers in the project and not a fault of the framework.
@@nickchapsas If I remember right, NBomber was initially an internal project for testing a gambling system. And they have some ridiculous number of request - like thousands per second. I would think that they would want to see pretty consistent results. On our meagre 30-50 request per second, I haven't noticed any significant inconsistencies. Though from the video I see that K6 gives much more detailed result log. In case of our project, we decided that we should stick to the C#. In my experience, it is much easier to get other engineers involved with whatever thing you are doing if it is in the same language as the main project. After working for 5 years on a project where development and autotesting were in two different languages, I would rather fight a herd of radioactive monkeys than support anything on Ruby.-)
Great info! I knew jMeter but I didn't like it too much. Now I can test a logic endpoints sequence in many ways. We'll be implementing this on my work. Thank you!
How do you define a real spike number? I mean looks that you choose a random high number, but how u determine a realistic spike for your scenario. THANKS
Where do you run these tests effectively? Because the machine config might give you different results, right? Does it make sense to run this locally on your laptop?
For any commercial system, you should ideally have an environment that mirrors production in terms of performance, but has different data stores. e.g. PRE-PRODUCTION. You could configure K6 as part of the deployment pipeline for your PRE-PRODUCTION environment so that you can easily see performance improvements or degradation over time. You could also set up the same thing in your QA pipeline. Although the actual numbers would be different to a production like environment, you could still get a sense of performance degradation or improvement.
Question: Do you have to run your test against your CI build? Or your staging? or your prod? How do you compensate the differences in hardware when running test local or in CI?
@@nickchapsas so, you run in your CI against your pre-production? Can the bandwidth of the CI limit the results? I mean, you are "emulating" a lot of different users from a single machine (the CI server)
@@ciberman CI would run TF or Pulimi to provision infrastructure specific for the test and servers to execute the tests against it. The CI agent wouldn't run the tests directly but indirectly
Can we control rps to be a constant? For example, I want 1500 requests per second to occur for 10minutes, 800 for 5 minutes.. I get that it depends on number of users and sleep, combinations, but not sure how to calculate what numbers I need to setup based on rps.. thank you very much for your time and answer if you have the time to answer!
Thanks for the awesome video! I have nodejs backend that is hosted in Cloud Run so should I load test locally or on Cloud Run? Do I have to load test all API routes individually or all as a batch like you did in the first stress test?
Hello Is it possible to perform correlation using the showcased tool? How can one store a response value from from api call to use in another one? Thanks.
Hi Nick, I am a performance test engineer I want to know that in Load testing my requirement are put the load around 200 rps not more thant 200rps so is there any technique of throughput controller as we seen in Jmeter or Can I increase more load during test running state like locust tool
Hi Nick, great video and very interesting topic. If this is part of your pipeline, let's say for production deployment. Do you target production or do you set-up a temporary environment or even a slot to load test against? Running load test on the actual production env could affect current users right? Also I wonder how you deal with authentication using JWT, can you do some automated refresh step? Since the token would expire running those hour long tests.
great video. One question though: this 'framework' handles just a simple endpoint. How does the project framework look like when we're supposed to handle 10 endpoints? Each endpoint with a separate Load.js, spike.js, etc? Or is it possible to use the same .js file for multiple endpoints; if yes, how does the code look like?Thanks
You can manage that in any way you want. The framework gives you the tools for both one or multiple endpoints either in batch or in sequence and you handle the full flow any way you want
Hi Nick ... Can you please tell me how to create server job to send email to all users without Api still waiting to complete sending all email ... By the way i am using SMTP from Microsoft account .... Thanks 😊
Would love to see how to integrate a stress test into the check-in process with multiple devs committing several times a day. Is the pattern to bottleneck on a single deployed environment? Or how do others manage to run tests where code needs to be deployed to validate a pull-request?
I think this is something that would need to run on a nightly schedule and not after each push. That way it does not block the build pipeline and you still get performance results daily. If needed you should also be able to run this type of build manually if there is a need to push something out to production and you want to ensure that there is no performance degradation before releasing any code. It all depends on what you want...
It would be awesome if you show how to test API with xUnit Framework. I know how to test with SQL server but testing with MongoDB in the same way seems impossible.
Thank you for this insanely simple and straight forward way to do load testing.
Very helpful! I like that Nick shows all these small and big things that sometimes difficult to find somewhere. One of my fav dev channels!
This is great! Would be great to see a similar video on how to go about understanding the limits(load test) of an event/message based application, if possible
I would actually watch the jmeter video too btw... I'm so in.
Such a good tutorial, leaving a comment for algorithm. Thanks mate
This is a great breakdown of how to get started with K6!
Great info, this is something I always intend to learn about but never know where to start. Thanks!
This video is just amazing🎉👏
Great video Nick!
so simply explained great job
Awesome! Really good video and tool.
Very well explained 👏
Never knew it was so simple to do, definitely gonna try this.
This is great, thanks! I’ll start using it today.
Thank you for simple and clear explanation.
Looks pretty good. Will definitely be testing this out.
What would be interesting is if you included an example of 'here is an example of an API that where stressing it caused it to fail.' Then apply some fix, run it again, and compare resulting metrics.
But great video/idea.
This video is a setup for a future video in which I will use a soak test to identify a memory leak in .NET. Just needed to have this one out first for reference.
@@nickchapsas makes sense!
Very nice tutorial. Thanks!
Thanks for this amazing video sir!
Hi Nick, thanks for this explanation
Thanks man! Thats what I needed.
Nice one thanks for sharing ! Yes need k6 on websites as well please. Thank you once again for this video
Nobody likes stress but we can always learn to manage it. Keep it up!
Awesome content Master Nick! Perfect
Nice video!
Tip: you can use CTRL+L instead of clear. Also there is recursive search feature CTR+R
Thankssss, this is just what i needed, Like and subcribed
Nice one. Exactly what I need to know right now.
Superb video !
One of the Bestttttttt video .
Thank you! This was very helpful. I'd love to see these applied in JMeter.
So cool bro, I have to try it
Great video, I will use it as a starting point for performance testing on my project!!!
Thank you very much for the excellent explanation!
nice and easy to understand video, pls also do a video with jmeter
Thanx man that was huge
Thanks will check K6. I usually use Jmeter but it is hard to maintain it.
This is great stuff :)
Yes please the load testing for web apps please Nick!!!
I'd love to see load testing integrated on a website. I am building more APIs than I used to but I still manage a good number of websites that don't consume a dedicated API.
nice. waiting for website testing video
hello nick, love your videos! maybe you can make a video about sonarQube and how it can integrate with your project and tests
We use NBomber. It is C# based (well, there is F# underneath). As I understand, it should have very similar functionality. But it is C#.
We were very reluctant to support anything in not-C# and NBomber covers most of our needs.
There was a bit of a struggle with implementing load tests. But in 9 cases out of 10 it was because of a lack of performance testers in the project and not a fault of the framework.
NBomber is significantly less consistent in my experience and less performant.
@@nickchapsas If I remember right, NBomber was initially an internal project for testing a gambling system. And they have some ridiculous number of request - like thousands per second. I would think that they would want to see pretty consistent results.
On our meagre 30-50 request per second, I haven't noticed any significant inconsistencies.
Though from the video I see that K6 gives much more detailed result log.
In case of our project, we decided that we should stick to the C#. In my experience, it is much easier to get other engineers involved with whatever thing you are doing if it is in the same language as the main project. After working for 5 years on a project where development and autotesting were in two different languages, I would rather fight a herd of radioactive monkeys than support anything on Ruby.-)
Very useful, thanks.
Great video, I would like to see a video for testing web apps like blazor server
Will be nice to see something similar for websites.
Great info! I knew jMeter but I didn't like it too much. Now I can test a logic endpoints sequence in many ways. We'll be implementing this on my work. Thank you!
yeah!
where can i download your code you just showed? for practice?
How do you define a real spike number? I mean looks that you choose a random high number, but how u determine a realistic spike for your scenario. THANKS
Fantastic
Where do you run these tests effectively? Because the machine config might give you different results, right? Does it make sense to run this locally on your laptop?
Any input about how k6 works with APIs that needs OAuth oidc based authentication will be great
You’d have to manage the flow as part of the test or have a stage where you authenticate the users before you start firing the tests
In a CD pipeline which environment would you run these tests against? Would you build a production like environment to run them?
For any commercial system, you should ideally have an environment that mirrors production in terms of performance, but has different data stores. e.g. PRE-PRODUCTION. You could configure K6 as part of the deployment pipeline for your PRE-PRODUCTION environment so that you can easily see performance improvements or degradation over time.
You could also set up the same thing in your QA pipeline. Although the actual numbers would be different to a production like environment, you could still get a sense of performance degradation or improvement.
Thanks Nick. Request to share video on mqtt load testing with X.509 certificate for .net.
Question: Do you have to run your test against your CI build? Or your staging? or your prod? How do you compensate the differences in hardware when running test local or in CI?
Usually for those tests to tell you something useful, you need to run them in environment that mirror's prod. That is usually pre-prod
@@nickchapsas so, you run in your CI against your pre-production? Can the bandwidth of the CI limit the results? I mean, you are "emulating" a lot of different users from a single machine (the CI server)
@@ciberman CI would run TF or Pulimi to provision infrastructure specific for the test and servers to execute the tests against it. The CI agent wouldn't run the tests directly but indirectly
Can we control rps to be a constant? For example, I want 1500 requests per second to occur for 10minutes, 800 for 5 minutes.. I get that it depends on number of users and sleep, combinations, but not sure how to calculate what numbers I need to setup based on rps.. thank you very much for your time and answer if you have the time to answer!
Thanks for the awesome video! I have nodejs backend that is hosted in Cloud Run so should I load test locally or on Cloud Run? Do I have to load test all API routes individually or all as a batch like you did in the first stress test?
Do you have the source code shared somewhere (github or something)? Thanks!
Hello
Is it possible to perform correlation using the showcased tool? How can one store a response value from from api call to use in another one? Thanks.
Awesome. Can we get on web applications
I would take a look at codes about entire stress_test. Where do I figure out this code?
Nice one
You seem to be predicting the problems I’m trying to solve.. lol I wonder where I’m going to find someone top notch when I recruit my dev team 😉
Pricey test for PROD on any cloud, you know ;)
Please do for websites/web applications also
can we handle the constant options in the command line for load testing
You can yeah
how about using 'West Wind WebSurge' tool?
How to authenticate the API with bearer token in K6 ..? Need help
How did you plot these graphs ?
So it's similar to that of Jmeter.
Hi Nick, you said you will be doing this type of video for a web application. Can we expect it soon?
I would just add a POST call to the video
Hi Nick, I am a performance test engineer I want to know that in Load testing my requirement are put the load around 200 rps not more thant 200rps so is there any technique of throughput controller as we seen in Jmeter or Can I increase more load during test running state like locust tool
Would you still recommend this now it's bought by Grafana?
I’d recommend it even more now
Thank you for the swift reply, I’ve found this video very helpful. Do you have any courses on this?
@@leahsmartie Not yet but I will
Hi Nick, great video and very interesting topic. If this is part of your pipeline, let's say for production deployment. Do you target production or do you set-up a temporary environment or even a slot to load test against? Running load test on the actual production env could affect current users right?
Also I wonder how you deal with authentication using JWT, can you do some automated refresh step? Since the token would expire running those hour long tests.
Link for JMeter video pls... if not pls make it
can you make a video for load testing for mobile apps like react native apps
Awesome video dude, keep going, like and subscribe!
great video. One question though: this 'framework' handles just a simple endpoint. How does the project framework look like when we're supposed to handle 10 endpoints? Each endpoint with a separate Load.js, spike.js, etc? Or is it possible to use the same .js file for multiple endpoints; if yes, how does the code look like?Thanks
You can manage that in any way you want. The framework gives you the tools for both one or multiple endpoints either in batch or in sequence and you handle the full flow any way you want
Please make one for Web with dependency injection please.
Hi Nick ... Can you please tell me how to create server job to send email to all users without Api still waiting to complete sending all email ... By the way i am using SMTP from Microsoft account .... Thanks 😊
Where is the code you are using in your videos?
It’s available to my Patreons
Would love to see how to integrate a stress test into the check-in process with multiple devs committing several times a day. Is the pattern to bottleneck on a single deployed environment? Or how do others manage to run tests where code needs to be deployed to validate a pull-request?
I think this is something that would need to run on a nightly schedule and not after each push. That way it does not block the build pipeline and you still get performance results daily. If needed you should also be able to run this type of build manually if there is a need to push something out to production and you want to ensure that there is no performance degradation before releasing any code. It all depends on what you want...
very cool, I cannot find your code.
do a jmeter please...
Oops, I thought this was another kind of API ... 😂 I was looking for API (American Petroleum Institute) but got shown this ...
haha it happens!
As far as I know target is virtual users. the question is - is virtual user = ordinar user from real life?
This depends on how you set it up. They can certainly reflect an ordinary user yeah
@@nickchapsas how it can be configured?
Please can you include the link to the code on your repository
The code is available to my patreons
It would be awesome if you show how to test API with xUnit Framework. I know how to test with SQL server but testing with MongoDB in the same way seems impossible.
Thanks for the amazing content Nick, would appreciate it a lot if you could create similar content for website as well.
Can you share the source code for this?
The source code is available to my patreons
👍🏽
This blows JMeter out of the waters
agreed!
Good video but I wish it was more practical (as implied) than theoretical.
Running the tests on local machine, won't give real results cause it depends on your computer. What we might do to make the tests more realistic
I never recommended that you run the tests locally. You run the tests against a pre-prod environment from a good server.
js load tests 🤣🤣🤣
It’s a js engine but the tests are run in Go
Great video Nick!