You have covered many in depth concepts of Kafka. I couldn't find how transaction is processed . Could you please create a video how to handle transactions in kafka
@KnowledgeAmplifier1 is there any chance in future will see apache flink tutorial . I am looking for this tutorial . Can you please let me know any good documents or videos are available for this course. Thanks advance
i can be wrong, but i feel like the two different concepts are being mixed? i felt like you suggested that idempotence is needed for avoiding duplicate, and max in flight is needed to be set to 1 to ensure ordering but i think with idempotence, the max in flight property is not needed to be set to 1 even for ordering in your example when you said kafka writes first batch, third batch and then the second batch, with idempotence, kafka would know the third batch is out of order so throw an exception, so max in flight can still be a high value. but i can be wrong, not sure
hi shameek , when the max_in_flight_requests_per_connection is set to 1 is stricly imposes ordering of the messages but it wont prevent duplicates in case of retries but when you enable idempotence it avoids duplicate. and regarding the last point you mentioned , if the value is set to higher it will throw the exception saying KafkaException and it that case we will have to handle DLQ so that this message is removed from the queue making the queue to fetch the new message.
![Idempotent Producer in Apache Kafka [Apache Kafka Tutorials #16]](/img/n.gif)
Very informative video
Glad you think so! Happy Learning :-)
Great job in explaining important components
Hello KLNP1978, Thank You ! Happy Learning
U R KAFKA GOD !
😅😄 Thank you for your comment Ayush Mandloi! Happy Learning
You dint explain how duplicacy check happens? That’s the most important question
You have covered many in depth concepts of Kafka. I couldn't find how transaction is processed . Could you please create a video how to handle transactions in kafka
Very good explanation 😃
Thanks a lot 😊
@KnowledgeAmplifier1 is there any chance in future will see apache flink tutorial . I am looking for this tutorial .
Can you please let me know any good documents or videos are available for this course.
Thanks advance
Hi sir, can you please post a video for matlab program for blood pressure Estimation from ppg signal using cnn
it is impossible that someone might have started you kafka series and by this point he/she would not have subscribed.
Thank you shreyas for your inspiring comment!
You dint explain how duplicacy check happens? That’s the most important question
i can be wrong, but i feel like the two different concepts are being mixed?
i felt like you suggested that idempotence is needed for avoiding duplicate, and max in flight is needed to be set to 1 to ensure ordering
but i think with idempotence, the max in flight property is not needed to be set to 1 even for ordering
in your example when you said kafka writes first batch, third batch and then the second batch, with idempotence, kafka would know the third batch is out of order so throw an exception, so max in flight can still be a high value. but i can be wrong, not sure
hi shameek , when the max_in_flight_requests_per_connection is set to 1 is stricly imposes ordering of the messages but it wont prevent duplicates in case of retries but when you enable idempotence it avoids duplicate.
and regarding the last point you mentioned , if the value is set to higher it will throw the exception saying KafkaException and it that case we will have to handle DLQ so that this message is removed from the queue making the queue to fetch the new message.
could u please start a series on Snowflake🙏🙏
You dint explain how duplicacy check happens? That’s the most important question