Thank you for another insightful video! I really appreciate the effort you put into breaking down the differences between the two systems. It’s always great to see deep dives into these technologies. However, I noticed a few points that could use some clarification. I think, you oversimplify by saying Kafka consumer groups lack state, and JetStream consumers are stateful. In reality, Kafka consumer groups do manage state, particularly offsets, which are stored in a highly available way within Kafka itself. So, consumer groups can resume processing from the correct position after failures. The second point is, you suggest that Kafka requires extensive planning for partitions and that partitioning is cumbersome. I'd agree that's true and partitioning needs careful consideration, but it is also Kafka’s primary mechanism for scaling horizontally. Proper partitioning is critical for Kafka’s ability to handle massive amounts of data efficiently. Also, Kafka provides tools to manage and rebalance partitions when necessary. Finally, you claim that Kafka offers limited options for message consumption compared to NATS JetStream, particularly in filtering and controlling replay behavior. Actually, Kafka provides significant flexibility through consumer offsets, compacted topics, and Kafka Streams, which allow for complex processing, filtering, and windowing of data. There're some other points to discuss, but these are most crucial. Cheers ;)
Agree this comparison is very biased unfortunately and just on the first 8 minutes makes a number of mistakes. Kafka actually allows both: consumers handling the state (offset) or the broker managing it. Also Consumer Groups (their state) live on the broker and not on the client (same as Consumers in NATS).In Kafka you can also consume from a specific point in time. Finally a consumer group in Kafka can also consume a set of topics. So plenty of “errors” here.
When is the next video coming up ? One of the key reasons Kafka is used is for absolute ordering of messages to a unique consumer from a group. This video ended right before that topic for NATS which is really one of the missing key points for NATS to replace kafka when dealing with realtime in-order events consumption distributed across multiple consumers from the same group.
Is it possible to set up NATS in Fan-out mode to efficiently stream events to multiple nodes of the same backend so that specific event is delivered to each node?
Most folks today are replacing Kafka in some way, shape or form. I think if you are running into friction with Kafka or feel like you need a team of people to babysit it, NATS would be a good candidate to replace it
Good information about NATS, but I found it biased in a way that if someone who doesn't know anything about Apache Kafka might say why on earth should I ever use Kafka. And how someone who is expert can make such mistake about offsets...
While I appreciate your perspective, I would argue that the information about NATS was presented objectively. the goal is not to discourage someone who is unfamiliar with Apache Kafka, but rather to highlight the unique attributes of NATS. Experts are not infallible, and while the point about offsets might appear to be a mistake, it is quite possible it reflects a differing understanding or experience with these systems.
Again, admiring the diagraming skills 🤯
Thank you for another insightful video! I really appreciate the effort you put into breaking down the differences between the two systems.
It’s always great to see deep dives into these technologies. However, I noticed a few points that could use some clarification.
I think, you oversimplify by saying Kafka consumer groups lack state, and JetStream consumers are stateful.
In reality, Kafka consumer groups do manage state, particularly offsets, which are stored in a highly available way within Kafka itself. So, consumer groups can resume processing from the correct position after failures.
The second point is, you suggest that Kafka requires extensive planning for partitions and that partitioning is cumbersome. I'd agree that's true and partitioning needs careful consideration, but it is also Kafka’s primary mechanism for scaling horizontally.
Proper partitioning is critical for Kafka’s ability to handle massive amounts of data efficiently. Also, Kafka provides tools to manage and rebalance partitions when necessary.
Finally, you claim that Kafka offers limited options for message consumption compared to NATS JetStream, particularly in filtering and controlling replay behavior. Actually, Kafka provides significant flexibility through consumer offsets, compacted topics, and Kafka Streams, which allow for complex processing, filtering, and windowing of data.
There're some other points to discuss, but these are most crucial.
Cheers ;)
Agree this comparison is very biased unfortunately and just on the first 8 minutes makes a number of mistakes. Kafka actually allows both: consumers handling the state (offset) or the broker managing it. Also Consumer Groups (their state) live on the broker and not on the client (same as Consumers in NATS).In Kafka you can also consume from a specific point in time. Finally a consumer group in Kafka can also consume a set of topics. So plenty of “errors” here.
Thanks for the clarification here. We will definitely clarify the assumption here in the video
When is the next video coming up ? One of the key reasons Kafka is used is for absolute ordering of messages to a unique consumer from a group.
This video ended right before that topic for NATS which is really one of the missing key points for NATS to replace kafka when dealing with realtime in-order events consumption distributed across multiple consumers from the same group.
Hello, thanks for the video. Please when will be there the part 3? I'm interested in the ordering of messages. Here you've mentioned it quite briefly.
Good point. Jean Noel and I need to do another video
Wrong, Kafka stores state (offsets) on a separate topic and these offsets are per partition and per consumer group, not on a client side
Thanks for the clarification, we’ll update the video to clarify the specifics here
Is it possible to set up NATS in Fan-out mode to efficiently stream events to multiple nodes of the same backend so that specific event is delivered to each node?
I'd like to know as well
hello greate video can you make one configuring with helm making jetstreams with helm infact using helm to do clusters and everything
anyone knows which software he is using to draw the diagrams? it's really cool
I'm using Excalidraw!
This is excellent video. Jetstream seems to have everything and more., does that mean we can replace kafka?
Most folks today are replacing Kafka in some way, shape or form.
I think if you are running into friction with Kafka or feel like you need a team of people to babysit it, NATS would be a good candidate to replace it
Good information about NATS, but I found it biased in a way that if someone who doesn't know anything about Apache Kafka might say why on earth should I ever use Kafka. And how someone who is expert can make such mistake about offsets...
While I appreciate your perspective, I would argue that the information about NATS was presented objectively. the goal is not to discourage someone who is unfamiliar with Apache Kafka, but rather to highlight the unique attributes of NATS. Experts are not infallible, and while the point about offsets might appear to be a mistake, it is quite possible it reflects a differing understanding or experience with these systems.
hello greate video can you make one configuring with helm making jetstreams with helm infact using helm to do clusters and everything