Significant Scalability Benefits in Spring Boot 3.2 using Virtual Threads
Вставка
- Опубліковано 29 вер 2024
- In this video we explore how to take advantage of Virtual Threads in your next Spring Boot Application. Virtual Threads offer significant scalability benefits in your Spring MVC applications that perform blocking calls such as talking to a database, writing to an input stream or talking to another service over HTTP.
🔗Resources & Links mentioned in this video:
Blog Post: www.danvega.de...
👋🏻Connect with me:
Website: www.danvega.dev
Twitter: / therealdanvega
Github: github.com/dan...
LinkedIn: / danvega
Newsletter: www.danvega.de...
SUBSCRIBE TO MY CHANNEL: bit.ly/2re4GH0 ❤️
Thanks for the great content.
It seems to be that the comparison is not 100% fair as you reduced from 200 default to 10 threads. So I guess the real difference is probably when you really hit the wall for the performance of the thread per request model. So I guess the more interesting would be to leave the default 200 threads setting and then spin up -n 1000 with less blocking time to show the real power.
Nevertheless, I think the the point you showed is great and very thankful. Thanks for very useful content!
In addition to removing the upper limit related to number of cores virtual threads also use less memory. 200 system threads will require around 200 MB of RAM
It should be more about memory. If built correctly, where you needed to scale 4 Microservices for some task, you can potentially get by w/ 2 etc as you're using less memory (assuming the underlying thing you're calling can handle it as well).
why this flag is not on by default?
Hello, mister Dan. That was a concise and easy to understand overview. But what is the trade-off behind enabling virtual threads? There definitely is a reason why this neat feature is not enabled by default. Should we all just switch out blocking MVC apps to virtual threads and live happily or there must be some consideration?
Nice explanation! Why did we turn off the tomcat.threads.max when we turned the virtual threads on?
It actually didn't matter...its either or, once you enable virtual threads that is no longer relevant. Nice catch
What about
ThreadLocal
and InheritableThreadLocal
is it still going to work wen i switch to virtual Threads
and what about MDC.put from slf4j
It still works with virtual threads, like it did with platform threads
Great explanation! Should come really handy in IO bound apps. Would this break Spring Security or other things that use ThreadLocal - for example when storing the SecurityContext?
With the introduction of virtual thread, several interfaces have been redesigned. As one of them, Thread Local can also be used in the traditional way. (I think it's the strength of language that runs on vm.) Instead, excessive use of Thread Local can overrun memory, and in fact, it eliminated a lot of use of TheadLocal in the implementation on the java.base package.
hey Dan, thanks for sharing! Being part of java 21 I could assume this is production ready right? great content!
Please make a video how to deploy spring boot app on aws lambda using graalvm
Your video overlay is hiding a significant portion of your presentation and its annoying
Sorry 🤷♂️
I like it, but migrating a jdk 17 app running in Docker requires jumping through a few hoops to enable jdk 21. Not insurmountable, but more than a single line in a config file.
All the same, I love the direction this is going. Thanks for the introduction.
Can you say more about what those hoops are? I expect to migrate some workloads to containers next year, likely without Java 21 at first.
I am curious about thos hoops too. You can you Buildpacks to automatically build the image from source and that comes ootb with the Spring Boot plugin, that makes it actually 0 lines of code.
@jamesStansell and @abelsalgadoromero4402,
As with *any* migration/upgrade it will take some effort. First there is the compatibility roulette, of finding what versions of the various plugins you are using work with JDK 21, and which will need to be upgraded. I then also found that for some, they were still under milestones, so I needed to bring in the spring-milestones repository. Finally, if you're just getting going with containers, figuring out how to upgrade the jdk in your container may take a bit of doing too.
As I said, not insurmountable, and the move to virtual threads is really great news.
I don't think @19:27, there was a need to comment out the server.tomcat.threads.max=10 property before running the example with virtual threads
Actually, keeping the max tomcat threads same would've validated the non-blocking nature of virtual threads. That is what I got when I executed virtual threads example with max tomcat threads as 10. Same result i.e. around 12sec
Quick question, on my computer I had max 200 threads and it ran. Anyone knows how this is possible? I have 8 cores on my computer.
Thank you very much Dan, i wanted to ask about failed requests 52, is it right?
Does it mean we don't need reactive libraries (like Reactor) anymore?? (If using spring boot)
Great question. If you were only reaching for reactive programming as a scalability mechanism you now have an option on the imperative side of the house. Reactive programming is still a great architecture choice for streams based processing and dealing with back pressure. Another JEP in Project Loom (Structured Concurrency) is still in preview that is something else Webflux solves for.
Ah right, I forgot about back pressure. So @@DanVega from your experience, what are some common problems that can be solved easily using reactive programming and that get complicated when doing imperative? I am thinking about maybe batch jobs, but even spring has Spring Batch right? I don't think it follows a reactive approach.
Great example! That extra bit of time is most prob due to the OS and some other overhead contending for share of the 10 threads. Pretty impressive.
Thank you for the video Dan! I have question about the virtual threads. At 19:03 I see "Failed requests: 5", and after you enabled the virtual threads at 20:53 I see "Failed requests: 52". Does this mean that most of the requests failed?
We can observe that they're failed req.s because of length mismatches, you can pass -l arg to ab to avoid. So we can ignore them in our case.
Thank you for that answer!
Thank you!@@nm69
bro just wanted to say that you're amazing and the things i learned on this channel is the absolute best, thanks to you king!!
Great! but what is the best practice is i want to enable Virtual Threads for single route in my application ? manually create and run virtual Thread in controller?
You don't enable virtual threads for a single endpoint you do it for your entire application. You don't need to manually create a virtual thread in your controller as Spring is handling that for you.
Hi Dan, was the max thread limit to 10 only to simulate a hardware (cpu core count) bottleneck? bcz I believe even if you didn't enable the virtual threads, the whole request would take similar time since your machine is capable of it even without the virtual threads. Am I correct?
I had that same concern, so I am sure you are right.
I think this depends on the type of the pool type. In case of fixed threadpool here matching with core count, for platform threads was meant to dictate don't create new thread for new request/task, beyond 10. Fixed thread pool has the strategy that till the cap is hit it will spawn new thread, unlike fork join pool. Usually setting more than 10 would pretty much be insignificant. After certain point the performance can vary because of context switches.
Do keep in mind that there is a limit to physical threads you can setup. The default with tomcat is 200 - so while this example did constraint it down to 10 artificially - the same example when scaled would produce comparable results. Virtual Thread can effectively work with no limits (or really high) so they have a lot more scaling than physical threads
Not sure why Dan changed max threads property. I have tested similar code with 300 concurrent threads, when virtual threads are turned on, the app becomes unresponsive. Without virtual threads, it just works fine.
@@htejwani1 that doesn't make sense 😒
Thanks for your great explanation, very easy to understand
It is very important and cool feature provided and very easy to plugIn. Superb explaination and demo @DanVega. Thank you so much.
You're very welcome!
Expected you to show the final console output to make sure not all 10 platform threads were taken. Does someone know what thread names would be printed?
You can run the example and see it in action but you should see the worker thread not go above 1.
Nice video, it would be cool if you help us understand if a billion request come in at once, there will be a memory issue. Can you force the server to only accept so many active request? I always wondered about that.
There are a few approaches here:
1. Do you want to serve all of those Billion requests? Then you need horizontal scaling, with your app servers sitting behind a load balancer, and preferably having auto-scaling enabled. If there's a DB involved, then you will also need to have multiple read-replicas set up, and a lot of caching between your app and the DB.
2. Do you want to discard requests that you can't serve? In that case, you can set up rate limiting. Any requests that breach your rate limit can be served with an HTTP 429 response (too many requests). They client would have to retry the request once traffic has died down.
Of course, there are many more ways of handling a billion requests, I've just mentioned these two from top of my head.
You forgot to show the output while running virtual threads!
You can do this on your machine if you want to give it a try.
Hey Dan, why you didn't limit platform thread for virtual thread testing on properties file as you did without virtual thread? So that we could understand that 10 platform thread concurrently handle 60 request without blocking themselves.
Great tutorial ❤
I have a question regarding how well a component scoped per request with @RequestScope would perform
From my tests, the reason why the timing measurements are never exact is because the 1st request is always treated separately, adding 3 extra seconds. So on video's minute 19:23, instead of the expected 18 seconds you get 21 seconds @DanVega.
Thank you very much it was very interesting and helpful. just want to add why there is an additional seconds in those tests, need to do warm up to the jvm and after that the benchmark will get more realistic and consistent.
Hey Dan - Nicely done on the video! Can't wait to try this out myself. Did you look at JVM memory consumption? I'm curious to see what, if any impact on memory there is. Thanks!
Good to hear from you friend 🤩 I haven't done any comparisons with JVM memory consumption but I like where your heads at. I will take a look into that.
@@DanVega - Let me know what you find. I'm curious from a operation cost perspective. ie running ~20 Spring Boot microservices in a K8 cluster. You can chew up a lot of memory quickly, which can drive up costs due to runtime memory needs.
Thank you !!!!
Great content with an excellent explanation, keep going Dan ;)
Glad I'm able to help out
is there a connection pool that is compatible with virtual threads yet ?
So, in short Virtual Threads is similar to what Kotlin is doing with coroutines, isn't it?
Nice video. Please what is the name of the software you used to make the slides?
That is a great tutorial! Very easy-to-understand examples and it clarified all the important concepts!
Hey Dan what about home office? What do you think?
What do you mean home office?
@@DanVega i heared that all vmware employees should get back to office. Somehow i cannot link the article.
still light years away from erlang model :)
Thank you for the video. I am building the url with query params.If pass the header then not able to pass the query params. If pass the query params then not able to pass the bearer token. How to pass the bearer token in the header?
I got it. Thank you
Great video! Thank you!
As always best explanation. keep it up Dan!
great explanation, thanks for share it!
You are amazing 🎉
Thanks for easy explanation ❤
You just made my day! Thank you for the kind words 🤩
Thanks a lot
Thank you!
Hi Dan, can you please tell me how you got the tshirt 😊?
I made a few polos for myself :)
@@DanVega pls tell me ,how I can get same . Please share link if you are fine.
14:06 It's a little off topic question, but what applications were used here?
thanks sir