Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/
I want to extend my sincere thanks for this amazing course. I have been following your kafka playlist and being a kafka-newbie, this course has been amazing. Again, many thanks for your efforts.
Thanks a lot for creating this playlist and making most of the people lives easy to understand the Kafka. I would say it is one of the finest explanations to understand the kafka starting from the scratch.
Thank you very much for your perfect explanation, which helps me a lot to understand the complex topic of partition assignment and revocation. I like your slow and clear voice, so that I could follow you without a problem. Keep up the good work.
I am working on Kafka 0.9.0 last 1 year and facing the problem of rebalancing by the coordinator while consumer process is no longer be running. While I restart consumer process it is taking rebalancing always. Now I have exactly understand how to commit offsets properly. Thanks a lot for sharing the valuable knowledge.👍🏻
Hi Sir, Thank you so much for the excellent explanation. I have few doubts. 1. Can rebalance possible in single partion, single consumer scenario. If possible do we need to implement rebalance listener for this scenario. 2. Can we statically bind that partition to that consumer. 3. Can we disable rebalance listener. Thank you very much sir 🙏
Hi,Thank you sir for sharing your knowledge.The way u take the concept and explaining is awesome. After watching your spark videos i got the confidence that i can face interviews with more confidence .Thanks for showing way how to do in real time using GCP. Now i started learning kafka parallel with spark. These videos are in terminal.Can you please explain kafka also in real time with GCP. That will be a great advantage for those who are moving to this ecosystem. Thank you
Sir Fantastic tutorial and superbly explained....One question Can we not commit partition passed in onPartitionsRevoked method to avoid commtfailedexception..?
Hi, This is a very nice series for Kafka.Probably the best. Thanks for putting it together. Wanted to get clarity - The commit in the consumer code is commented out. Is it because you want to show how rebalancing works? So had two questions for you 1. How the commit will happen when next loop comes around. Would you uncomment the consumer.commitSync(.......) ? 2. Do I need to write additional code in the rebalance listener to clear out the offset map. My understanding would be that the Map would get overwritten. Please suggest. Thanks Viren
Greate work Sir!!!!. This is great material is saving Tons of money to people who will have to other wise take stupi course in Edureka or other sites :)
Superb! I think this is THE BEST tutorials on KAFKA. When I searched few months back, there were no this clear recordings. Thanks for getting this to the community. I have one question. Can we configure how many records Consumer can read in one poll ? I mean say there are 1000 records at the broker level and when you say number '20' records returns in your example, I hope that is a max-configured one? Like several producer configs, can you please explain different kind of polls at the consumer side?
We try to control max number of records by putting various limits like max.poll.records and passing a timeout parameter to poll method. The poll method tries to get you as many as possible within those limits.
Hi, thanks for the great video. I have the following question here: you are giving an example of how to make processing of a single message and commit an offset as a single atomic operation. This is easily achieved in your case since you are storing data in the same database. So you are utilizing JDBC transaction. What should I do if I have to consume a message and then I am supposed to POST it to the external API instead of saving it in the database? If my POST to the API was successful and my consumer crashed between storing offset information, I think my message might be processed twice.
Thanks Mr. LJ (unfortunately I haven't figured out your name yet :( ) for the great tutorials on Kafka. I have a query which is bothering me for a while. Is it a good practice to use the following for processing of records: 1. Use JMS to which Consumer can drop messages (records from Kafka Broker) for an actual end application to process gracefully. That way one could atleast commit offsets without the need to wait long ? 2. Use another Kafka Broker (may be a separate instance for processing of records). So basically pull stream of data from a broker and push to another broker. I know sounds crazy but is it used anywhere ?
Let me understand your question.1. You want to use JMS as a buffer because you think processing message will take a long time and we have to wait for that before we commit. Are you sure? I guess, it will complicate the solution and I don't see a reason. 2. I guess you mean pull from one topic, process it and push to another topic ( you didn't mean - from one broker to another broker). I wonder, why would you need another Kafka Cluster.
Thanks sir. I guess I get when you say JMS would complicate the solution as we are ideally going from distributed messaging to single point messaging. As for point 2, it was just a wild thought. However do we have topic to topic routing in Kafka ?
One small doubt sir, Is re-balancing activity local to the consumer group? Ex. Suppose we have 2 consumer gropus CG1 & CG2 and If consumer gets added/removed to/from CG1, it will trigger re-balancing only for CG1 or it will trigger re-balancing for CG2 also PLEASE REPLY ASAP, thanks alot GREAT TUTORIALS, hatsoff!
And this re-balancing won't happen if I add a producer which starts writing to the same topic from which a consumer group is reading (no addition/subtraction of consumer when producer got added) ?
Hi, I have a situation. I have multiple consumers on same topic. One of consumer received 10 events to process. After processing 6 event all of the sudden consumer goes down. I am using auto commit. Now my problem is what is status of 4 items which is in consumer queue. Waht happens to current and commit offset. How can stop data loss situation here. Please help me. For ur inform we using Azure Function Kafka Trigger.
Hi Sir.. Thanks for this tutorial. I have one doubt , How we will do it if we dont know the partition no... what would be the addoffset() here ..can we write TopicPartition only with topic argument
Once onPartitionsRevoked() call back is received, I understand we are trying to commit the offset. I think we should also stop processing the remaining messages, am i correct ? Suppose during the last poll() we received 100 messages and at the time of processing 51th message we received onPartitionsRevoked() callback and we would call commitSync that we processed 50 messages so far. Messages from 51 through 100 may get processed by the newly assigned consumer, won't that end up in duplicate processing of records? Thanks
7 років тому
at 8.18, in addOffset method, at the end, you write new OffsetAndMetadata(offset,"commit"); what does it mean? why you put a string "commit"?
That's metadata. It could be any string,. You can use any non-null string instead of Commit.
7 років тому
And what's the purpose? You're putting in the hashmap with key topicpartition a value of Offset and Metadata, right? but I don't understand where you use it and what could be useful for. Thank you :)
For starting KAFKA we need to start zookeeper.but in cloudera VM i think there is zookeeper instance already running .SO can anyone tell me how to check through VM terminal whether the default instance is runnig or not ? Also if we start a zookeeper service we can see the logger screen but how to check it for cloudera VM.
Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.
www.learningjournal.guru/courses/
I want to extend my sincere thanks for this amazing course. I have been following your kafka playlist and being a kafka-newbie, this course has been amazing. Again, many thanks for your efforts.
Thanks a lot for creating this playlist and making most of the people lives easy to understand the Kafka. I would say it is one of the finest explanations to understand the kafka starting from the scratch.
Thank you very much for your perfect explanation, which helps me a lot to understand the complex topic of partition assignment and revocation. I like your slow and clear voice, so that I could follow you without a problem. Keep up the good work.
This is fantastic, loved your presentation! Also this video cleared my previous doubt...Thanks a million!!!
Amazing videos, way you present to audience, the flow is unbeleivable i cannot thank you more
I am working on Kafka 0.9.0 last 1 year and facing the problem of rebalancing by the coordinator while consumer process is no longer be running. While I restart consumer process it is taking rebalancing always. Now I have exactly understand how to commit offsets properly. Thanks a lot for sharing the valuable knowledge.👍🏻
This is a fantastic tutorial, I've been looking for this explanation for some time. Thanks
Simply awesome video explaining the rebalancing
Best tutorial I found so far. Thank you so much Sir for sharing your knowledge.
Very clear and concise explanation of Kafka!! I cannot wait for other tutorials. Thanks!!
Thanks a lot for the feedback, I am working and committed to at least one every week.
Brilliant explanation - thanks
@Learning Journal Shouldn't we use ConcurrentHashMap as multiple consumers will try to update the map simultaneously?
good job in explaining kafka in this way - straight to the point I like it :)
Excellent Explanation
Hi Sir, Thank you so much for the excellent explanation. I have few doubts.
1. Can rebalance possible in single partion, single consumer scenario. If possible do we need to implement rebalance listener for this scenario.
2. Can we statically bind that partition to that consumer.
3. Can we disable rebalance listener.
Thank you very much sir 🙏
These vids are amazing. You rock!
Great video
This is really a nice tutorial to start with kafka, thanks for all your effort
Thank you sir for a practical explanation with a problem statement and a resolution.
Sir you have perfectly represented and explained all.
Hi,Thank you sir for sharing your knowledge.The way u take the concept and explaining is awesome. After watching your spark videos i got the confidence that i can face interviews with more confidence .Thanks for showing way how to do in real time using GCP.
Now i started learning kafka parallel with spark. These videos are in terminal.Can you please explain kafka also in real time with GCP. That will be a great advantage for those who are moving to this ecosystem. Thank you
Sir Fantastic tutorial and superbly explained....One question Can we not commit partition passed in onPartitionsRevoked method to avoid commtfailedexception..?
Hi,
This is a very nice series for Kafka.Probably the best. Thanks for putting it together.
Wanted to get clarity - The commit in the consumer code is commented out. Is it because you want to show how rebalancing works? So had two questions for you
1. How the commit will happen when next loop comes around. Would you uncomment the consumer.commitSync(.......) ?
2. Do I need to write additional code in the rebalance listener to clear out the offset map. My understanding would be that the Map would get overwritten.
Please suggest.
Thanks
Viren
Your Complete tutorial is very good and clear to follow. Thanks a lot for sharing us this info :)
Greate work Sir!!!!. This is great material is saving Tons of money to people who will have to other wise take stupi course in Edureka or other sites :)
Superb! I think this is THE BEST tutorials on KAFKA. When I searched few months back, there were no this clear recordings.
Thanks for getting this to the community.
I have one question. Can we configure how many records Consumer can read in one poll ? I mean say there are 1000 records at the broker level and when you say number '20' records returns in your example, I hope that is a max-configured one?
Like several producer configs, can you please explain different kind of polls at the consumer side?
We try to control max number of records by putting various limits like max.poll.records and passing a timeout parameter to poll method. The poll method tries to get you as many as possible within those limits.
Hi, thanks for the great video. I have the following question here: you are giving an example of how to make processing of a single message and commit an offset as a single atomic operation. This is easily achieved in your case since you are storing data in the same database. So you are utilizing JDBC transaction.
What should I do if I have to consume a message and then I am supposed to POST it to the external API instead of saving it in the database? If my POST to the API was successful and my consumer crashed between storing offset information, I think my message might be processed twice.
Great work sir.... God bless you..
Thank you!
Thanks Mr. LJ (unfortunately I haven't figured out your name yet :( ) for the great tutorials on Kafka. I have a query which is bothering me for a while. Is it a good practice to use the following for processing of records:
1. Use JMS to which Consumer can drop messages (records from Kafka Broker) for an actual end application to process gracefully. That way one could atleast commit offsets without the need to wait long ?
2. Use another Kafka Broker (may be a separate instance for processing of records). So basically pull stream of data from a broker and push to another broker. I know sounds crazy but is it used anywhere ?
Let me understand your question.1. You want to use JMS as a buffer because you think processing message will take a long time and we have to wait for that before we commit. Are you sure? I guess, it will complicate the solution and I don't see a reason.
2. I guess you mean pull from one topic, process it and push to another topic ( you didn't mean - from one broker to another broker). I wonder, why would you need another Kafka Cluster.
Thanks sir. I guess I get when you say JMS would complicate the solution as we are ideally going from distributed messaging to single point messaging.
As for point 2, it was just a wild thought. However do we have topic to topic routing in Kafka ?
Hi Sir, your Kafka teachings are fantastic. Can you please teach on Spring Kafka. Thank you Sir 🙏
One small doubt sir, Is re-balancing activity local to the consumer group? Ex. Suppose we have 2 consumer gropus CG1 & CG2 and If consumer gets added/removed to/from CG1, it will trigger re-balancing only for CG1 or it will trigger re-balancing for CG2 also
PLEASE REPLY ASAP, thanks alot
GREAT TUTORIALS, hatsoff!
Yes, You got it correct.
@@ScholarNest thanks alot prashant sir
so your saying the kafka can either be a producer/sender or consumer/reciever?
Very good explanation..awesome..
Thanks a lot for your encouragement.
Can u pls explain how can we achieve the same when @kafkalistner is used
Thanks a lot !! Excellent tutorial ..
And this re-balancing won't happen if I add a producer which starts writing to the same topic from which a consumer group is reading (no addition/subtraction of consumer when producer got added) ?
Rebalancing has nothing to do with the producer.
I am writing a service to retrieve past messages of a topic and trying to show.Can you please help me which approach is better choice for this.
Thank you.
How to calculate number partitions are required for topic??
Hi, I have a situation. I have multiple consumers on same topic. One of consumer received 10 events to process. After processing 6 event all of the sudden consumer goes down. I am using auto commit. Now my problem is what is status of 4 items which is in consumer queue. Waht happens to current and commit offset. How can stop data loss situation here. Please help me.
For ur inform we using Azure Function Kafka Trigger.
Hi Sir..
Thanks for this tutorial.
I have one doubt , How we will do it if we dont know the partition no...
what would be the addoffset() here ..can we write TopicPartition only with topic argument
Ok...So We can not create topic without partitions
Sir how to clear offset value. pls help me
Once onPartitionsRevoked() call back is received, I understand we are trying to commit the offset. I think we should also stop processing the remaining messages, am i correct ? Suppose during the last poll() we received 100 messages and at the time of processing 51th message we received onPartitionsRevoked() callback and we would call commitSync that we processed 50 messages so far. Messages from 51 through 100 may get processed by the newly assigned consumer, won't that end up in duplicate processing of records? Thanks
at 8.18, in addOffset method, at the end, you write new OffsetAndMetadata(offset,"commit"); what does it mean? why you put a string "commit"?
That's metadata. It could be any string,. You can use any non-null string instead of Commit.
And what's the purpose? You're putting in the hashmap with key topicpartition a value of Offset and Metadata, right? but I don't understand where you use it and what could be useful for. Thank you :)
For starting KAFKA we need to start zookeeper.but in cloudera VM i think there is zookeeper instance already running .SO can anyone tell me how to check through VM terminal whether the default instance is runnig or not ? Also if we start a zookeeper service we can see the logger screen but how to check it for cloudera VM.
your tuto are excellent. let me know how I can reward you . tipee?
Your appreciation is my reward.