It would be cool to see what the Ntex framework (Rust) can do. It’s what the original creator of Actix moved-on to after giving Actix over to the community. It seems like it performs really well on the benchmarks I’ve seen it in, but it would be cool to see it in this format here against the other rust frameworks!
Honestly the fact that you took feedback and went back to make further comparisons that incorporated the feedback from different communities may make it it one of the best videos of this kind around.
Thank you for correcting your mistake by rerunning the tests with optimisations! Mistakes are completely normal, but seeing corrections is surprisingly rare...
@@CoolestPossibleName there is some truth to that though. not that it doesn't have benefits over c, but i don't know if the benefits are strong enough for most companies to justify it over c. odin is kind of in that same place. in my opinion, c3 has more compelling arguments going for it.
Zig has only a safe mode to detect faults at runtime, in real word application you would release optimised for space or speed without the runtime failsafe
Release Safe does not ensure (full) memory safety. It adds stuff like UB checks (some of it is missing in Rust), but no "full" memory safety like in rust.
As a Rust dev, its really cool to see Zig and Rust trade blows. considering picking up Zig where i would normally reach for C due to the dev environment.
even after those optimization performance and compile command change, rust still on the top. its so satisfying how the developer and man behind rust and actix can achive such performance.... they are the GOAT
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0g Yep. Its cool that its so fast, but I would expect a systems language to have a worse DX than Go. But I think choosing feature complete framework in any language (e.g. Laravel) is also a very good solution. No "stack" hopping
@@NabekenProG87DX comes down to preference. I found golang panicked too easily and the error handling was pretty bad. Rust's pattern matching, union types (enums) and macros make it a joy to work with if you can push through the steep learning curve. But it comes down to preference in the end
@@ionutale1950Depends wholly on the application. In-mem caching can help a lot with db hits. If you’re using a global state you can batch appends and read state once per server on ticks and it’d stay pretty light. If you’re displaying tables too big for cache for numerous reports and clients then yeah you’ll have some DB pains. But you likely wouldn’t get that kind of volume for that kind of thing.
rust is a low level lang that doesn't feel low level at all, also it will not allow you to use pointers like crazy, and forces you to write everything thread safe, u can only make shitty code if u use "unsafe" keyword. also "thread_local" macro feels like magic, its wonderful.
@@Cuca-hn3md I wouldn't call Rust a low level language. It is a compiled language that allows you to write extremely efficient code and provides memory safety despite manual memory management. Those are indeed two highly desirable traits in system software, but that doesn't make Rust a low level language. Go, with its relatively low expressiveness, is a low level language - most of what you can write in other languages will take at least one and a half as many lines of code in Go. Rust, however, doesn't have this problem. With Rust, you can write a database driver or a Web application equally well, without writing much more code than if you'd use languages traditionally used for each job. In fact, if you compete against plain C in the database driver, chances are the Rust code will be smaller. I'd venture to say Rust be about on par with Python for the Web application.
@@a0flj0 "level" could also mean how much control you have over what will actually happen in the computer. The fact that you can write inline assembly in Rust makes it IMO as low level as C.
I'd like to see a comparison between the same web app on Rust Actix-Web and Elixir Phoenix. It might look like apples and oranges but it'll be interesting to see the memory utilization of the two apps. Elixir "processes" are light threads. Actix implements its own light threads but I understand those still use more memory than Elixir (BEAM) processes
Actix-web stopped using the Actix actors, tho. And yes, Rust actors cannot be as efficient as BEAM actors, because BEAMs are preemptible (need assembly for that) and all state in BEAM is immutable.
@@LtdJorge Thank you. I wasn't aware of the Actix-web change. I still think the comparison would be interesting to see though. It'll put specific numbers to the memory usage
Note on the previous benchmark: Zig was in 'debug' mode, which is the default, and I didn’t know that. I found a "production-ready" Dockerfile and thought it was properly compiled, so I’m not the only one who made that mistake.
Rust can actually be FASTER than Zig, C or C++ in some cases because their ownership and aliasing rules for &T and &mut T allow for more aggressive optimization techniques. Although the downside to Rust lifetimes is that it restricts which programs the compiler will mark as semantically correct, that is also one of its benefits. Since Rust programs are restricted to a smaller subset, the compiler can also make more aggressive assumptions/optimizations of those valid programs.
Unfortunately there are a lot of cases where the assembly generated by LLVM is suboptimal, like additional memory/register copies. It would be very interesting to see a Rust compiler that is built from the ground up to take advantages of all available guarantees.
You are talking about avoiding unnecessary copy operations? Does Rust actually have an advantage over Zig here? Zig does seem to do rather aggressive optimizations in this department already. Since Zig is going to have its own backend, it might be able to benefit more from Zig specific optimizations in the future. For the time being, most effort in Zig seems to go into optimizing the compiler performance (incremental compilation, etc.). I'm curious to see what happens when the focus shifts to code generation and optimization later on.
@@michaelutech4786 If Zig uses pointers, it should be the exact same. I suspect iterators are the main reason why. The Zig/C implementation probably does bounds and null checks every iteration of a loop. Rust's iterators don't have to.
Hi @AntonPutra I've been following your channel for a while now and I absolutely love your content on comparing tests and benchmarks of different programming languages and frameworks. I was wondering if you could consider doing a similar comparison between in-memory databases like Redis and RabbitMQ. I'm particularly interested in seeing how they perform under various workloads and use cases. Thanks for all the great content!
Thank you! Actually, the second benchmark should have been Redis vs. Memcache, but every time I get distracted by reading all the comments. I'll get there eventually 😂
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
150k with 2 cores that is a lot! i did some similar tests, but on my local machine with NestJs and Go Fibber, not even close: - NestJs 400 req/s - Go Fibber around 10k dam, rust is fast. i am wondering if there is anything faster than Rust, like a server is C or C++
I mean they would roughly be the same performance, since they are "manage your own memory" languages. what differs is the different HTTP framework implementation which actix seems to have the edge
Zig is something like 0.6 version lang m8. Id say that was pretty fucking impressive how zig ran that C library. Only small margin behind fully fledged rust. Id say that Zig will literally eat C
@@ionutale1950I would normally argue for it just being a bad zig implementation, but with the weird behavior we see, I suspect that most Zig server frameworks are just C copies, and a lot of C frameworks are pretty stable and reliable, but outdated when it comes to algorithms and implementations.
Yep, have only just started on this Will need to duplicate your test setup and iterate through config options to get it scaling properly for starters Current implementation that caps out like you saw uses only 1 worker thread :) So need to have some playtime with these AWS boxes to see how they behave under load
It's the difference between caching a lot and no caching at all. Caching means availability remains constant as long as you can keep caching requests at the cost of using more memory. Actix is almost completely functional; no caching.
Thank you for this comparison, can you take into account the content of the queries? Actix sends more Data ( Http headers ) so you need Bytes/sec metrics. Also it would be really interesting to change Serde by Sonic-rs which is more performant, which will make Rust win even more points. And why not use Hyper directly instead of Actix? Thanks again 🙏🏻
What was the request output? Could you make it more "real world". Like an json response of 1kb, 10kb, 100kb. And then another test that compiles some zig to html or rust to html (using thr same 1kb, 10kb, and 100kb html outputs). Could you also add golang to these ?
@sighupcmd A proposal for a more practical benchmark: add two dimensions, JSON/API and HTML/Templating. This moves away from pure theory towards real-world use cases. Research into optimal engines for each will be needed, trading some purity for more relevant metrics.
@@Matty0187 again, you propose to bench libs responsible for json/html (where are a dozens of them, btw). What's the point of this here? Here author used "minimal" required stack to make service respond. That's the correct way. Also, there's only one real use case: yours (your service's).
@@sighupcmd errors, rps, and latency while great metrics, a response of "hello world" from a server is a naive. We should measure bytes or packets out as an additional dimension. I still wouls love to see how common template and JSON responses effect the rps, latency, and error rates cross the servers
Probably a fair result then. The “std” zig implementation there is using only 1 thread, so 80k shows us where the hardware is getting saturated The flatline from there on is (likely) kubernetes throttling that maxed out single CPU Not surprisingly- all 3 implementations that are using the same llvm optimizer here perform very close up to that point. @anton - send me some links plz to how to duplicate your test setup so we can properly tune this for the environment it’s running on. Need to iterate the config to find a suitable balance here Fun project! Working with low level code (be it C / Rust / Zig whatever) is like tuning a car to match the course it’s racing on :) At least the audience is enjoying the show :)
haha, sure I'm using EKS which is a managed Kubernetes cluster in AWS. If you have never used it before it can be problematic and expensive. You can use minikube locally but you would never be able to reproduce the benchmark locally. But at least you can test it and compare it. So minikube - minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download Kubernetes deployments - github.com/antonputra/tutorials/tree/main/lessons/208/deploy you can apply with kubectl apply -f ... also i have clients as well - github.com/antonputra/tutorials/tree/main/lessons/208/1-test also kubectl apply -f ... all images including clients are compiled for both arm and amd, so you can run them on mac as well if you need any help please let me know
@@AntonPutra Cheers thanks. Yeah, I will go back and review some of your playlists, and try out those deploy scripts. Managed EKS is probs the way to go then. I dont mind spending a $bit to get it right .. just have to remember to turn it off when im done :) Interested in finding the code for your load-tester container - is that in a lesson somewhere ?
@@steveoc64 i have older version written in go with prometheus metrics - github.com/antonputra/tutorials/tree/main/lessons/201/client but it is really outdated but can give you an idea. i recently rewrote that client in rust but it is not ready for release yet. with go client i used m7a.8xlarge instances with new rust client i only need 4xlarge
@@steveoc64 Also, make sure when you run those tests in the cloud that your application and client are deployed in the same availability zone. AWS has data transfer charges between regions and between availability zones, but inside the same zone, it is free. If you start running load tests between zones, it could cost thousands of dollars in data transfer fees, so be careful.
Cool to see that actix performance is basically the same as what is essentially a C implementation. Also you managed to show clearly what the effect is of setting cpu limits in kubernetes. When it starts throttling the stdlib one you see the latency jump up and the amount of requests it can handle plummet. Would be interesting to see what would happen if you run the exact same test but on a m7a.large without cpu limits. I would expect it to then not have such a sudden breaking point but just gradually increase latency with sustained max requests per second
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
@@AntonPutra Ah I see what you mean, cgroups have cpu shares which is what requests map to (after calculations). Just running in k8s without limits should already give the insights since all nodes will have the same kubelet overhead
Exactly what sort of hardware does m7a.large provide? Are they quoting real cores, or are they including hyperthreaded cores ? It does make a bit of a difference depending on what the app does Would be interesting as well to compare x86 vs Arm offerings on AWS
How do you time-lapse thru the results so it looks like they are going really fast? At first I thought it was real time and then I realized you are replaying them somehow on grafana
I don't agree with the "only use Rust if you need the performance" since I prefer the DX of Rust over Golang. Once I was used to the ownership concept and wrote some async code, I don't feel like Rust is slowing me down at all. Although the problem that if one function becomes async and the async keyword has to propagate to all caller functions is pretty annoying. Learning Rust was probably easier for me since I come from a C/C++ background, I guess it's much harder to learn Rust if you only used garbage collected languages before.
Would be cool to see this in comparison to older languages like C(++) and java, that might not be using the newer techniques, but do have years of optimizations
Honestly with how much more knowledge we have now, it would be very surprising if you can "optimize away" the difference. It will still be C at the top unless there is a paradigm shift (like the functional style of Actix).
As a Rust dev, I was expecting more from zig on all fronts in this test, as I keep hearing that zig is more performant and has better memory control, while being somewhat memory safe. For it to not beat rust in any of these fields really showed me that any low system programming will accumulate a die hard, cult like, core, that will try to make a sell for things the language cannot even still do. In this case, zig is still far too young to compete with a powerhouse like Actix, and once mature, probably will match it more than beat it. Thats fine. The true selling point of the language is to be c like to ease dev in it and to improve over c. Rust is at the extreme opposite and you might fit in its targeted audience or not, but regardless, it delivers on the core premises of memory safety at no performance cost. In this regard, taking sides seems pointless. As a dev, you are not your tool, you just use the tools that you have at your disposal and are the most expert with even if its not the greatest one for the task (I look at you nodeJs and Flask). So learn both, become an expert on the one you prefer and respect those who prefered the other option.
the pure zig implementation is another league on memory usage, it's heavily optimized to avoid heap allocations as much as possible at the expense of using the heap for parallelism with that memory usage it can scale an order of magnitude better than both zap and actix actix is an overall winner here, it has a mature and more featured api, memory usage & req/s better than zap zap is little immature as a project, it will benefit from both code optimization & from zig future versions
this hit rock bottom. golang for fast prototyping and changing requirement. rust for rewriting core business or legacy system which cost is too expensive
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0g sadly, i am currently working in a company with very little care of prototyping. once it's implemented in python / node, it will stay that way. make sure your company can afford rewrite. mine is not that friendly toward this. that's why prototyping in golang is better, consider that we are only talking about web server in this video. python is a trap for complex web server. nodejs is better but we know "javascript"
yes, zig was in 'debug' mode, which is the default, and i didn’t know that. i found a production-ready dockerfile, and i thought it was properly compiled, so i'm not the only one who made that mistake.
The Zig standard library sucks. And its documentation sucks even more. That's not, unfortunately, something new. The standard lib has always been an afterthought and will still be the case as Andrew Kelley doesn't seem to care about it.
well, I can only help if someone provides a well-optimized version, but again, performance is not the main criteria for why people choose one language over another
not really, i start with 20 independent client instances that generate load, then i increase the number of concurrent users for each instance by 1 every 30 seconds. so i start with 20 truly independent users and by the end of the test i run 4000
I think the most impressive thing is that the Zig implementation were written by very small team if any, in a very short amount of time, in a language that's still very new. This to me feels like the real achievement of Zig, being fairly simple to use and write good code in. Rust is amazing, but it really takes some serious time to become productive in it, it takes even more time to understand how to write efficient Rust code. Early one I used to delay the difficulty by .clone() everything and .? everything or using RC everywhere. I think this confirm my intuition, Zig is going to be, if it keep on becoming more and more popular, the go to language for high performance software where safety is not necessarily the absolute main concern. Rust is going to remain the best choice to rewrite critical software in. I like this diversity, I think Rust is one of the best language to "Rewrite stuff" but not really a good language to explore, so that makes perfect sense.
It uses LLVM just like Rust, performance limit is pretty much the same until in-line assembly. I’d definitely expect performance hits to be a result from copying outdated C implementations vs creating custom implementations, which is what I expect the weird memory behavior is coming from too.
Zig is a pleasant language that is a joy to program in. It feels proper language, its approach to memory safety is more correct than Rust making it way simpler. I still not a big fan shorthand syntax but I can live with it. Zigg feels like a sensible refinement of C language for system programming. I like it, but clearly work is still needed on its stdlib for better implementations, being native compiled they should not see differences unless there is some inefficiencies in the library implementation .
We should compare 0.6 version rust from 2016 to that 0.6 version Zig. Never ever i have written not a single line of Zig, but from halfway done lang compared fully fledged battle tested lang that was very impressive how fast Zig ran that C library. Id say Zig will literally eat C
👋What should I test next???
👉 [Playlist] New Benchmarks: ua-cam.com/play/PLiMWaCMwGJXmcDLvMQeORJ-j_jayKaLVn.html&si=p-UOaVM_6_SFx52H
node vs bun vs deno please
kotlin native vs kotlin jvm vs go vs expressjs vs hono
Express vs Laravel vs Django vs Fastapi vs Fastify
c vs c++ vs rust vs go
It would be cool to see what the Ntex framework (Rust) can do. It’s what the original creator of Actix moved-on to after giving Actix over to the community. It seems like it performs really well on the benchmarks I’ve seen it in, but it would be cool to see it in this format here against the other rust frameworks!
Honestly the fact that you took feedback and went back to make further comparisons that incorporated the feedback from different communities may make it it one of the best videos of this kind around.
thanks, i always read every single comment and improve with each new video and fix mistakes
Thank you for correcting your mistake by rerunning the tests with optimisations! Mistakes are completely normal, but seeing corrections is surprisingly rare...
yeahh, Anton is cool
always!
How fucking good that not even 1.0 zig took the C library and ran it like a fucking champ. Zig has great future ahead
Hardly. It has no benefits over C.
@@KushLemon You clearly haven't used zig enough
@@KushLemon*It has an ever-so-slight benefit over C23. It has significant benefits over C17/C99/ANSI
Not sure if this is serious or bait lmao
@@CoolestPossibleName there is some truth to that though. not that it doesn't have benefits over c, but i don't know if the benefits are strong enough for most companies to justify it over c. odin is kind of in that same place. in my opinion, c3 has more compelling arguments going for it.
I would like to see this compiled with Zig safe considering that is the biggest selling point of rust is memory safety.
Zig has only a safe mode to detect faults at runtime, in real word application you would release optimised for space or speed without the runtime failsafe
well i did compiled zig in safe mode in the previous benchmark 😊
Release Safe does not ensure (full) memory safety. It adds stuff like UB checks (some of it is missing in Rust), but no "full" memory safety like in rust.
@@AntonPutra Ah, I see! Thanks for letting me know 😅
As a Rust dev, its really cool to see Zig and Rust trade blows. considering picking up Zig where i would normally reach for C due to the dev environment.
🤭
even after those optimization performance and compile command change, rust still on the top. its so satisfying how the developer and man behind rust and actix can achive such performance.... they are the GOAT
fr
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0g Yep. Its cool that its so fast, but I would expect a systems language to have a worse DX than Go. But I think choosing feature complete framework in any language (e.g. Laravel) is also a very good solution. No "stack" hopping
@@Z3U5.0g imagine Go with memory arenas (like zig) ❤
@@NabekenProG87DX comes down to preference. I found golang panicked too easily and the error handling was pretty bad. Rust's pattern matching, union types (enums) and macros make it a joy to work with if you can push through the steep learning curve. But it comes down to preference in the end
FYI: Zap pre-allocates memory during startup, expecting higher loads… so memory tests don’t mean much.
true
100k req/s - like a few small towns clicking in app at the same time
Me when I forget to turn off my autoclicker on that one shitty Browser game ... on 20 machines
exacly, but with that load, you need like 16 vCPU for the database, and i may not be even close
@@ionutale1950Depends wholly on the application. In-mem caching can help a lot with db hits. If you’re using a global state you can batch appends and read state once per server on ticks and it’d stay pretty light. If you’re displaying tables too big for cache for numerous reports and clients then yeah you’ll have some DB pains. But you likely wouldn’t get that kind of volume for that kind of thing.
@@houstonbova3136 really? cache can do that?
wow, this must be 2024, where cache is being used
that's possible when it comes to a trading system.
Not today Zig 🤣, but we know you're younger.
wdym by "we".
Soon, accepting new PRs😊
Rust is such a beast, will learn it asap for side projects kkkkk
Try out Loco (based on rust) for a Ruby on Rails like setup that helps you get started, or Jeremy Chone on youtube with Rust 10x :)
rust is a low level lang that doesn't feel low level at all, also it will not allow you to use pointers like crazy, and forces you to write everything thread safe, u can only make shitty code if u use "unsafe" keyword.
also "thread_local" macro feels like magic, its wonderful.
@@Cuca-hn3md I wouldn't call Rust a low level language. It is a compiled language that allows you to write extremely efficient code and provides memory safety despite manual memory management. Those are indeed two highly desirable traits in system software, but that doesn't make Rust a low level language. Go, with its relatively low expressiveness, is a low level language - most of what you can write in other languages will take at least one and a half as many lines of code in Go. Rust, however, doesn't have this problem. With Rust, you can write a database driver or a Web application equally well, without writing much more code than if you'd use languages traditionally used for each job. In fact, if you compete against plain C in the database driver, chances are the Rust code will be smaller. I'd venture to say Rust be about on par with Python for the Web application.
It's nice, but it's quite unique and takes time to learn
@@a0flj0 "level" could also mean how much control you have over what will actually happen in the computer.
The fact that you can write inline assembly in Rust makes it IMO as low level as C.
I'd like to see a comparison between the same web app on Rust Actix-Web and Elixir Phoenix. It might look like apples and oranges but it'll be interesting to see the memory utilization of the two apps. Elixir "processes" are light threads. Actix implements its own light threads but I understand those still use more memory than Elixir (BEAM) processes
Actix-web stopped using the Actix actors, tho. And yes, Rust actors cannot be as efficient as BEAM actors, because BEAMs are preemptible (need assembly for that) and all state in BEAM is immutable.
@@LtdJorge Thank you. I wasn't aware of the Actix-web change. I still think the comparison would be interesting to see though. It'll put specific numbers to the memory usage
ok noted!
Rust proved it's top, i don't care about zig, elixir, go or anything else.
@@iulikdev ok
Note on the previous benchmark: Zig was in 'debug' mode, which is the default, and I didn’t know that. I found a "production-ready" Dockerfile and thought it was properly compiled, so I’m not the only one who made that mistake.
Actix / Rust still rock hard !
Zig / Stdlib seem to have great memory usage.
it was optimized that way specifically for the test
Rust can actually be FASTER than Zig, C or C++ in some cases because their ownership and aliasing rules for &T and &mut T allow for more aggressive optimization techniques.
Although the downside to Rust lifetimes is that it restricts which programs the compiler will mark as semantically correct, that is also one of its benefits. Since Rust programs are restricted to a smaller subset, the compiler can also make more aggressive assumptions/optimizations of those valid programs.
Unfortunately there are a lot of cases where the assembly generated by LLVM is suboptimal, like additional memory/register copies. It would be very interesting to see a Rust compiler that is built from the ground up to take advantages of all available guarantees.
I think the most optimizations come from iterators in this case. There probably is a whole lot of bounds and null checking in the C/Zig libs.
@@dark0sv i think zig has a plan to remove dependency from llvm
You are talking about avoiding unnecessary copy operations? Does Rust actually have an advantage over Zig here? Zig does seem to do rather aggressive optimizations in this department already. Since Zig is going to have its own backend, it might be able to benefit more from Zig specific optimizations in the future.
For the time being, most effort in Zig seems to go into optimizing the compiler performance (incremental compilation, etc.). I'm curious to see what happens when the focus shifts to code generation and optimization later on.
@@michaelutech4786 If Zig uses pointers, it should be the exact same. I suspect iterators are the main reason why. The Zig/C implementation probably does bounds and null checks every iteration of a loop. Rust's iterators don't have to.
Hi @AntonPutra
I've been following your channel for a while now and I absolutely love your content on comparing tests and benchmarks of different programming languages and frameworks.
I was wondering if you could consider doing a similar comparison between in-memory databases like Redis and RabbitMQ. I'm particularly interested in seeing how they perform under various workloads and use cases.
Thanks for all the great content!
Thank you! Actually, the second benchmark should have been Redis vs. Memcache, but every time I get distracted by reading all the comments. I'll get there eventually 😂
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
Would love to see more golang stuff! Thanks for all the content anyway :)
thanks! django vs go next :)
It's time for c vs c++ vs rust vs go
Do people actually use C for web servers?
@@justsomeguy8385 use case might be there for low latency applications not sure still would like to see a comparison if possible
just couple more and i get to c++ i promise 😊
well i may use facil.io directly in c without zig wrapper
@@AntonPutra ... subscribed 🙂
150k with 2 cores that is a lot!
i did some similar tests, but on my local machine with NestJs and Go Fibber, not even close:
- NestJs 400 req/s
- Go Fibber around 10k
dam, rust is fast.
i am wondering if there is anything faster than Rust, like a server is C or C++
I mean they would roughly be the same performance, since they are "manage your own memory" languages. what differs is the different HTTP framework implementation which actix seems to have the edge
@@OrtinFargo zig sopposed to "roughly be the same" or even better, but is not. we should test and let the data decide
Zig is something like 0.6 version lang m8. Id say that was pretty fucking impressive how zig ran that C library. Only small margin behind fully fledged rust. Id say that Zig will literally eat C
@@rasvatissi580 i think you are right. zig is great
@@ionutale1950I would normally argue for it just being a bad zig implementation, but with the weird behavior we see, I suspect that most Zig server frameworks are just C copies, and a lot of C frameworks are pretty stable and reliable, but outdated when it comes to algorithms and implementations.
Good to see Rust remains competitive. I wonder what causes rust to drop out on the latency test though. Sobering weird with an allocator?
The memory usage gives it away. Actix is very functional. The processor overhead difference is due to caching.
yes it's caching
Looks like an opportunity for improvement!
🔴IMPORTANT🔴 I am still accepting PRs to improve Zig applications. As soon as I get significant improvements in Zig, I'll release an updated video!
Yep, have only just started on this
Will need to duplicate your test setup and iterate through config options to get it scaling properly for starters
Current implementation that caps out like you saw uses only 1 worker thread :)
So need to have some playtime with these AWS boxes to see how they behave under load
And... subscribed! This intellectual honesty is why.
Some things are weird here.
The availability drops of the rust app and the memory use of the native zig app.
It's the difference between caching a lot and no caching at all. Caching means availability remains constant as long as you can keep caching requests at the cost of using more memory. Actix is almost completely functional; no caching.
The native Zig app is 1Mb which isn't much different from the Rust app.
correct
Summon C++ magic developers to handle twice as many requests as Rust. 😄
will see :)
Thank you for this comparison, can you take into account the content of the queries? Actix sends more Data ( Http headers ) so you need Bytes/sec metrics.
Also it would be really interesting to change Serde by Sonic-rs which is more performant, which will make Rust win even more points.
And why not use Hyper directly instead of Actix?
Thanks again 🙏🏻
@@DevRJPro make the pr
ok noted, yes it really sends more data and i actually have network usage metrics, i may show it next time
пожалуйста сравни go с node js, gin/fiber vs nest.js/next.js + простые запросы к бд, может пулл соединений, очень интересно!
horosho, obyazatelno sdelau! zapisal
What was the request output? Could you make it more "real world". Like an json response of 1kb, 10kb, 100kb. And then another test that compiles some zig to html or rust to html (using thr same 1kb, 10kb, and 100kb html outputs). Could you also add golang to these ?
adding json parsing: that will additionally bench json parser lib performance, lowering the quality of "pure" stack bench
interesting, I'll see what i can do to make it short
@sighupcmd A proposal for a more practical benchmark: add two dimensions, JSON/API and HTML/Templating. This moves away from pure theory towards real-world use cases. Research into optimal engines for each will be needed, trading some purity for more relevant metrics.
@@Matty0187 again, you propose to bench libs responsible for json/html (where are a dozens of them, btw). What's the point of this here? Here author used "minimal" required stack to make service respond. That's the correct way.
Also, there's only one real use case: yours (your service's).
@@sighupcmd errors, rps, and latency while great metrics, a response of "hello world" from a server is a naive. We should measure bytes or packets out as an additional dimension. I still wouls love to see how common template and JSON responses effect the rps, latency, and error rates cross the servers
I love these followups!
thanks i do my best
Also i do want to see the Ntex vs Actix!!!!
haha, ok soon!
@@AntonPutra yes please!
This just makes me more in awe of Actix.
i was told that Ntex is even better
@@AntonPutra Making an Ntex vs Actix video would be interesting
@@almuhajer6760 will do, i'll add axum as well cause last time i didn't generate enough load
nice work
thank you!
love this series Anton!
thank you!
Thank you. Great job
thank you!
Nodejs vs php. I am interested in learning new insight on how to optimize both of them.
Exactly. I have been requesting this too. But he seems too focused on Rust :(
i imagine v8 would outperform php
thank you! will do!
Who use php in 2024?
Could you do a comparative with Actix and Spring or some Java framework?
ok noted!
Probably a fair result then. The “std” zig implementation there is using only 1 thread, so 80k shows us where the hardware is getting saturated
The flatline from there on is (likely) kubernetes throttling that maxed out single CPU
Not surprisingly- all 3 implementations that are using the same llvm optimizer here perform very close up to that point.
@anton - send me some links plz to how to duplicate your test setup so we can properly tune this for the environment it’s running on. Need to iterate the config to find a suitable balance here
Fun project!
Working with low level code (be it C / Rust / Zig whatever) is like tuning a car to match the course it’s racing on :)
At least the audience is enjoying the show :)
haha, sure I'm using EKS which is a managed Kubernetes cluster in AWS. If you have never used it before it can be problematic and expensive. You can use minikube locally but you would never be able to reproduce the benchmark locally. But at least you can test it and compare it.
So minikube - minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download
Kubernetes deployments - github.com/antonputra/tutorials/tree/main/lessons/208/deploy
you can apply with kubectl apply -f ...
also i have clients as well - github.com/antonputra/tutorials/tree/main/lessons/208/1-test
also kubectl apply -f ...
all images including clients are compiled for both arm and amd, so you can run them on mac as well
if you need any help please let me know
@@AntonPutra Cheers thanks. Yeah, I will go back and review some of your playlists, and try out those deploy scripts.
Managed EKS is probs the way to go then. I dont mind spending a $bit to get it right .. just have to remember to turn it off when im done :)
Interested in finding the code for your load-tester container - is that in a lesson somewhere ?
@@steveoc64 i have older version written in go with prometheus metrics - github.com/antonputra/tutorials/tree/main/lessons/201/client
but it is really outdated but can give you an idea. i recently rewrote that client in rust but it is not ready for release yet. with go client i used m7a.8xlarge instances with new rust client i only need 4xlarge
@@steveoc64 Also, make sure when you run those tests in the cloud that your application and client are deployed in the same availability zone. AWS has data transfer charges between regions and between availability zones, but inside the same zone, it is free. If you start running load tests between zones, it could cost thousands of dollars in data transfer fees, so be careful.
Next is nim vs c++
noted!
Cool to see that actix performance is basically the same as what is essentially a C implementation.
Also you managed to show clearly what the effect is of setting cpu limits in kubernetes. When it starts throttling the stdlib one you see the latency jump up and the amount of requests it can handle plummet. Would be interesting to see what would happen if you run the exact same test but on a m7a.large without cpu limits. I would expect it to then not have such a sudden breaking point but just gradually increase latency with sustained max requests per second
I was thinking of running this benchmark on a standalone VM, and I even have everything ready, including the systemctl service files. However, the problem is that when you run something without limits, it starts to affect all other services on that VM. For example, I use node-exporter to collect basic metrics, so without limits, those exporters and agents begin to degrade as well, causing gaps in your monitoring system. You can set cgroup limits using systemctl, but it would be similar to how Kubernetes handles it.
@@AntonPutra do you have cpu requests set for the other services? That should also give them their share of cpu
@@Kavantix in k8s yes (for example cadvisor/kube state metrics etc), on VM usually you don't set limits
@@AntonPutra Ah I see what you mean, cgroups have cpu shares which is what requests map to (after calculations). Just running in k8s without limits should already give the insights since all nodes will have the same kubelet overhead
Exactly what sort of hardware does m7a.large provide? Are they quoting real cores, or are they including hyperthreaded cores ?
It does make a bit of a difference depending on what the app does
Would be interesting as well to compare x86 vs Arm offerings on AWS
Can you also do Python? would be cool to see how much slower it is as compared to these compiled languages
yes django is next
can you do a actix vs django rest?
django is next
Amazing. 🎉
thanks!
Common Rust W
yes for now
Spring boot with virtual threads please
yes soon
Cool! may be you can examine the MOJO language?
Compare C++ too . I want to see something
up
soon!
How do you time-lapse thru the results so it looks like they are going really fast? At first I thought it was real time and then I realized you are replaying them somehow on grafana
oh just using editing software to speed up
I don't agree with the "only use Rust if you need the performance" since I prefer the DX of Rust over Golang. Once I was used to the ownership concept and wrote some async code, I don't feel like Rust is slowing me down at all. Although the problem that if one function becomes async and the async keyword has to propagate to all caller functions is pretty annoying.
Learning Rust was probably easier for me since I come from a C/C++ background, I guess it's much harder to learn Rust if you only used garbage collected languages before.
thanks for the feedback
Would be cool to see this in comparison to older languages like C(++) and java, that might not be using the newer techniques, but do have years of optimizations
Honestly with how much more knowledge we have now, it would be very surprising if you can "optimize away" the difference. It will still be C at the top unless there is a paradigm shift (like the functional style of Actix).
I'll try. Any language is only as good as the DevOps engineer running it in production😂
Ничего себе акцентище. Даже без перевода все понятно )
staraus :)
Zig in debug mode is compared to Rust optimised. Embarrassing, delete the video.
Run in Docker! What a troll.
debug mode???
github.com/antonputra/tutorials/blob/main/lessons/208/zap-app/Dockerfile#L15
AWESOME, i think axum would perform a little better here. And could you compare JAVA framework vs node ones?
well based on the previous benchmark Axum is a bit slower, but i'll do java soon
As a Rust dev, I was expecting more from zig on all fronts in this test, as I keep hearing that zig is more performant and has better memory control, while being somewhat memory safe.
For it to not beat rust in any of these fields really showed me that any low system programming will accumulate a die hard, cult like, core, that will try to make a sell for things the language cannot even still do. In this case, zig is still far too young to compete with a powerhouse like Actix, and once mature, probably will match it more than beat it.
Thats fine. The true selling point of the language is to be c like to ease dev in it and to improve over c.
Rust is at the extreme opposite and you might fit in its targeted audience or not, but regardless, it delivers on the core premises of memory safety at no performance cost.
In this regard, taking sides seems pointless. As a dev, you are not your tool, you just use the tools that you have at your disposal and are the most expert with even if its not the greatest one for the task (I look at you nodeJs and Flask).
So learn both, become an expert on the one you prefer and respect those who prefered the other option.
Unfortunately seems like most comments are about "winning'. This is the first time I've ever seen this phenomenon in programming
thanks for the feedback. as a DevOps, I try to stay neutral and fair to all the languages I deal with on a day-to-day basis
Cool!!!! i will try it myself
👍
Actix is wild
it is :)
At least we all can agree in something, CPP sucks...
@@AntonPutra C++
haha thanks, long day :)
maybe benchmark python vs go vs rust? To showcase the "tiers of performance"?
php (laravel) vs other.. why.. because in my country .. we have cult that php is best web app ...
haha, ok will do!
the pure zig implementation is another league on memory usage, it's heavily optimized to avoid heap allocations as much as possible at the expense of using the heap for parallelism
with that memory usage it can scale an order of magnitude better than both zap and actix
actix is an overall winner here, it has a mature and more featured api, memory usage & req/s better than zap
zap is little immature as a project, it will benefit from both code optimization & from zig future versions
As usual great content :)
thank you!
Can you expand on the 50% CPU usage autoscaling? I think it's an interesting detail.
sure, here is an example - github.com/antonputra/tutorials/blob/main/lessons/181/1-hpa/cpu-memory/k8s/2-hpa.yaml#L20
this hit rock bottom. golang for fast prototyping and changing requirement. rust for rewriting core business or legacy system which cost is too expensive
Python/Nodejs for fast prototyping. Golang for production 98% of the time. Rust less than 2% of the time for certain super low latency requirements like HFT.
@@Z3U5.0gYeah. Golang is the 98% use case language. Rust is only if you absolutely need it or have to because of concerns where golang can't be used
@@Z3U5.0g sadly, i am currently working in a company with very little care of prototyping. once it's implemented in python / node, it will stay that way. make sure your company can afford rewrite. mine is not that friendly toward this.
that's why prototyping in golang is better, consider that we are only talking about web server in this video.
python is a trap for complex web server. nodejs is better but we know "javascript"
yes, I just rewrote my clients in Rust to switch from 8xlarge to 4xlarge
try Fast API vs node js Vs php laravel
ok noted!
Could you add axum in the benchmark? Would make a lot of sense to compare it to actix.
i'll make another one actix vs axum vs ntex
Really nice video! Could you please compare actix with ntex framework on rust?
yes, soon! i was told ntex is the best
@@AntonPutra yeah, I read that it is from the creator of actix. Will be waiting!
⬇ Team Rust button
🤣
⬇️ Team Rust Sub-Button
Common W for Rust
Hell Yeah
They tried so hard😂😂😂😂😂 they even bringed a "specialist"😂
I tried as well, maybe I should have tried Graviton 🤔
Rust literally destroys the competitors
well, as soon as I can, I'll prove otherwise😂
@@AntonPutra Waiting 😄
LOL, so the last benchmark was optimized rust vs zig with NO OPTIMIZATIONS? xDD
yes, zig was in 'debug' mode, which is the default, and i didn’t know that. i found a production-ready dockerfile, and i thought it was properly compiled, so i'm not the only one who made that mistake.
Can we please please please finally have Kotlin Ktor vs Java vs Dotnet
ok soon!
Rust is awesome
it is
Zig bros malding in the chat
i tried 😔
Tips 🚭❌
Teeps ☣️✅
??
Crab 🦀 still wins 😭
@@ibrahimmohammed3484 Always! 🦀❤️
i tried
rust's still on top
well yes, i tied my best
The Zig standard library sucks. And its documentation sucks even more. That's not, unfortunately, something new. The standard lib has always been an afterthought and will still be the case as Andrew Kelley doesn't seem to care about it.
well, I can only help if someone provides a well-optimized version, but again, performance is not the main criteria for why people choose one language over another
please do php swoole
ok noted!
I like your video!! just wondering why your name is like indonesian name
i like Bali, just changed it 🤣
@@AntonPutra was there any improvement on C# , from the suggestions
@@leulgirma yes i got a few PRs, i'll cover it in the near future
@@AntonPutra whahaha mee too!
I need rust to fail
sure, please send me PR
Why?
@@fabiopetrillo because some people liked to see the world burn (though i dont like rust but had to use on tauri applications)
Keep dreaming. This won't happen! 🦀❤️
@@RustIsWinning the higher you fly the harder you fall
can someone explain 151K req/s. Does it mean 151k users
Yes, unless one user is artificially making multiple requests per second
in one second it can respond to 151k api requests/call (can be made by 151k users or maybe by one automating )
not really, i start with 20 independent client instances that generate load, then i increase the number of concurrent users for each instance by 1 every 30 seconds. so i start with 20 truly independent users and by the end of the test i run 4000
Common rust W
yes
Cool rust
yeap :)
Rust WITH SAFETY beats ZIG with NO SAFETY :)
Indeed.
Previous Benchmark - ua-cam.com/video/VxW0ijXAfOs/v-deo.html
Do a laravel vs express vs django benchmark. Battle of the interpreted
Zig diehards from last video's comments in shambles when their full memory control optimized solution doesn't outperform the Rust abstractions
Not making fun of Zig, it's a pretty cool language with still lots to grow. Just memeing the overly aggressive comments on the last video
I think the most impressive thing is that the Zig implementation were written by very small team if any, in a very short amount of time, in a language that's still very new. This to me feels like the real achievement of Zig, being fairly simple to use and write good code in. Rust is amazing, but it really takes some serious time to become productive in it, it takes even more time to understand how to write efficient Rust code. Early one I used to delay the difficulty by .clone() everything and .? everything or using RC everywhere. I think this confirm my intuition, Zig is going to be, if it keep on becoming more and more popular, the go to language for high performance software where safety is not necessarily the absolute main concern. Rust is going to remain the best choice to rewrite critical software in. I like this diversity, I think Rust is one of the best language to "Rewrite stuff" but not really a good language to explore, so that makes perfect sense.
It uses LLVM just like Rust, performance limit is pretty much the same until in-line assembly. I’d definitely expect performance hits to be a result from copying outdated C implementations vs creating custom implementations, which is what I expect the weird memory behavior is coming from too.
i tired and still accepting PRs!
Zig is a pleasant language that is a joy to program in. It feels proper language, its approach to memory safety is more correct than Rust making it way simpler. I still not a big fan shorthand syntax but I can live with it. Zigg feels like a sensible refinement of C language for system programming. I like it, but clearly work is still needed on its stdlib for better implementations, being native compiled they should not see differences unless there is some inefficiencies in the library implementation .
take the L
Honestly, skill issue. I find rust more appealing.
@@u-k I simply love working with Rust. Took me like 3 years but I'm there
Hell no. I will rather write C++ over zig anyday.
Can't age with you. I find Rust programming more joy.
I don't like the way how Zig trying to be secure.
meh
😔
This test just blow the bubble of zig.
🎉🎉Rust🎉🎉
I hope this video will be removed by any influence.
We should compare 0.6 version rust from 2016 to that 0.6 version Zig.
Never ever i have written not a single line of Zig, but from halfway done lang compared fully fledged battle tested lang that was very impressive how fast Zig ran that C library. Id say Zig will literally eat C
@@rasvatissi580 zig current version is 0.13 i think its still not prod ready version even though bun TigerBeetle uses it
i tried and still accepting new PRs
No matter how optimized zig is, rust will always beat zig
i tried
Ask internet about: VPP/HostStack/LDP/nginx
?