Each partition ensures the order of the messages contained therein. The producer can target messages to be produced onto a specific partition by the message key. Consumers can target a specific partition by specifying the key of the messages they're interested in. If the message order doesn't matter as in when transactions are idempotent(i.e. Order has shipped) then you can produce those messages in a round robin fashion to multiple partitions and have each partition assigned a consumer where each consumer belongs to the same consumer group, consume those messages at scale. 10 partitions then define 10 consumers within the same consumer group so that messages are consumed at scale of 10. Got it. The MOST important thing to know is if msg Order IS important for message consumption(i.e. CRUD event notification). If it is then you can only have 1 partition and have multiple consumer groups each with 1 consumer reading from that partition. If msg order IS NOT important then you can scale.
Very important aspect indeed. Consumer scalability is not elastic when message ordering matters. I think though in some cases one might sacrifice consistency for performance and scalability by using compensation transactions after processing instead at the Kafka consumer groups level in order to account for order, but it’s not always possible.
I'd add that even if message order is important to keep, you might still be able to scale your consumer group. If ordering doesn't necessarily matter across your entire log, but by some logical grouping of subsets of messages. I.e. I have a Topic with CRUD events from Customers A and B, and for each customer i want to preserve message order, but between customers i don't really care. Then I can for example pick "customerId" as my message key (or if necessary, implement some custom partitioning logic), ensuring that connected events where I actually do care about ordering will land on the same partition, thus be consumed in order, and still have 2 partitions with 2 consumer instances. Continuing the example it would be e.g.: - Partition 0 holds the events of Customer A - in order, and - Partition 1 holds the events of Customer B - in order. And since partitions are distributed between instances of a consumer group, one of my consumer instances will process the events of Customer A - in order, and the other one will process the events of Customer B still in order. In such a scenario you can have multiple consumer instances in the same group, while still retaining sufficient ordering for the messages.
Hello, great video. I'm trying to find a strategy to resolve transactions of microservices communicated to each other by Kafka. Do you think that is possible? Thanks for your help.
Great video! Any tips of: 1) how to define and monitor a good number of partitions? 2) how to handle consumers failures? For example, how to not get stucked because of a miss-formed message?
I guess you already figured everything out as you asked two years ago, but if you need a web console for managing/monitoring topics/brokers/clusters then provectuslabs/kafka-ui is a very simple. open source and easy to install solution. With docker it will take few minutes to install and run.
I have a question, because of two instances in the same consuming apply (group) will be assigned to differences of partition right ? so could i it miss messages from partition it not assigned ?
A consumer in a consumer group is definitely assigned to a particular partition. The ideal scenario is that the number of consumers in a consumer group is less or equal to the number of partitions so that there are no consumers being in idle state.
Even I have same doubt, if multiple instances are consuming, and getting messages from different partitions, won't that miss the messages from the partition from which it is not distributed?
We run Kafka consumer on Kubernetes. For one use case we use pod-name as the groupId to filter out specific events. Is having 1000s of groupId for a topic an issue? will the Kafka cluster cleanup unused groupid safely in the background (metadata and offsets)?
How do we scale(on what criteria) consumers in consumer group(considering we have 3 partition with 1 consumer to start with )? is it message count in topic and auto scaler decides to add in new consumer service ?
Hello Sir, We have an enterprise license for Confluent kafka. But the problem is, we want to integrate with SoftwareAG Webmethods version 10.15. We are looking for any documentation or videos about this. But there are no such thing as a best practice how to use it. Only documentation for Installation adapter and user guide without any example. Can you please provide a documentation for a simple usecase? thank you sir.
Can I have the consumer act as a producer(publisher) as well? I have a user case where a consumer (C1) polls a topic and pulls messages. after processing the message and performing a commit, it needs to notify another process to carry on remaining work. Given this use case is it a valid design for Consumer (C1) to publish a message to a different topic? i.e. C1 is also acting as a producer
I like the way Tim explains , straight to the brains
Each partition ensures the order of the messages contained therein. The producer can target messages to be produced onto a specific partition by the message key. Consumers can target a specific partition by specifying the key of the messages they're interested in. If the message order doesn't matter as in when transactions are idempotent(i.e. Order has shipped) then you can produce those messages in a round robin fashion to multiple partitions and have each partition assigned a consumer where each consumer belongs to the same consumer group, consume those messages at scale. 10 partitions then define 10 consumers within the same consumer group so that messages are consumed at scale of 10. Got it.
The MOST important thing to know is if msg Order IS important for message consumption(i.e. CRUD event notification). If it is then you can only have 1 partition and have multiple consumer groups each with 1 consumer reading from that partition. If msg order IS NOT important then you can scale.
Very important aspect indeed. Consumer scalability is not elastic when message ordering matters. I think though in some cases one might sacrifice consistency for performance and scalability by using compensation transactions after processing instead at the Kafka consumer groups level in order to account for order, but it’s not always possible.
What is the importance of key schema from consumer perspective? Is key important for the consumer within the application?
Interesting point about applying ordering guarantees by forcing only 1 partition to be present. Thank you!.
I'd add that even if message order is important to keep, you might still be able to scale your consumer group. If ordering doesn't necessarily matter across your entire log, but by some logical grouping of subsets of messages.
I.e. I have a Topic with CRUD events from Customers A and B, and for each customer i want to preserve message order, but between customers i don't really care.
Then I can for example pick "customerId" as my message key (or if necessary, implement some custom partitioning logic), ensuring that connected events where I actually do care about ordering will land on the same partition, thus be consumed in order, and still have 2 partitions with 2 consumer instances.
Continuing the example it would be e.g.:
- Partition 0 holds the events of Customer A - in order, and
- Partition 1 holds the events of Customer B - in order.
And since partitions are distributed between instances of a consumer group, one of my consumer instances will process the events of Customer A - in order, and the other one will process the events of Customer B still in order.
In such a scenario you can have multiple consumer instances in the same group, while still retaining sufficient ordering for the messages.
Wow - a totally perfect video, clearing most of my doubts in less than 7 minutes!
Thank You!
Hello, great video. I'm trying to find a strategy to resolve transactions of microservices communicated to each other by Kafka. Do you think that is possible? Thanks for your help.
Great video! Any tips of:
1) how to define and monitor a good number of partitions?
2) how to handle consumers failures? For example, how to not get stucked because of a miss-formed message?
I guess you already figured everything out as you asked two years ago, but if you need a web console for managing/monitoring topics/brokers/clusters then provectuslabs/kafka-ui is a very simple. open source and easy to install solution. With docker it will take few minutes to install and run.
Best Kafka educational videos out there. thank you
Tim has explained kafka `simplified` exceptionally well.
I have a question, because of two instances in the same consuming apply (group) will be assigned to differences of partition right ?
so could i it miss messages from partition it not assigned ?
A consumer in a consumer group is definitely assigned to a particular partition. The ideal scenario is that the number of consumers in a consumer group is less or equal to the number of partitions so that there are no consumers being in idle state.
Even I have same doubt, if multiple instances are consuming, and getting messages from different partitions, won't that miss the messages from the partition from which it is not distributed?
We run Kafka consumer on Kubernetes. For one use case we use pod-name as the groupId to filter out specific events. Is having 1000s of groupId for a topic an issue? will the Kafka cluster cleanup unused groupid safely in the background (metadata and offsets)?
How do we scale(on what criteria) consumers in consumer group(considering we have 3 partition with 1 consumer to start with )? is it message count in topic and auto scaler decides to add in new consumer service ?
Hello Sir, We have an enterprise license for Confluent kafka. But the problem is, we want to integrate with SoftwareAG Webmethods version 10.15. We are looking for any documentation or videos about this. But there are no such thing as a best practice how to use it. Only documentation for Installation adapter and user guide without any example. Can you please provide a documentation for a simple usecase? thank you sir.
where i can bought that t-shirt! it's so cool!
Where can I get hands on api examples
Great videos!
Hi I have a doubt in this partition
Can I have the consumer act as a producer(publisher) as well? I have a user case where a consumer (C1) polls a topic and pulls messages. after processing the message and performing a commit, it needs to notify another process to carry on remaining work. Given this use case is it a valid design for Consumer (C1) to publish a message to a different topic? i.e. C1 is also acting as a producer
use kafka streams
Very nice
Number of consumers is restricted to number of partitions?
Number of consumers in a consumer group is limited to number of partition. You can have multiple consumer group consuming from one partition
Complicated!!!