Very good explanation vinod. Really helpful for the guys who want to learn kafka. Please share real time scenarios/issues related to kafka. Good job!!!
I have a question, if kafka assigns each partition to a consumer in a consumer group, and each partition hold unique messages of a topic. What happens if a consumer 1 needs messages from a topic it has subscribed to, and that message resides in a partition that is assigned to another consumer 2 instead?
All consumers belonging to each consumer group should work towards a particular goal. Imagine that a twitter account has 4 tweeted 4 tweets and u want to insert these 4 tweets into a database. These 4 tweets are stored in a topic of 2 partitions. Each partition has 2 tweets. Now, if u have one consumer in ur consumer group, this one consumer will consume all the 4 tweets sequentially from each partition and then inserts all the 4 tweets into database; In case, u have two consumers, each consumer will consumer 2 tweets from each paritition in parallel and insert them into DB. In both the cases, the end goal of inserting 4 tweets to database is achieved. This is how consumers in a consumer group work. They split the work towards achieving a common goal. In your case, as long as consumer 2 consumes the message, consumer 1 should not care because both consumer 1 and consumer 2 should work towards a common goal. In case, you have a different goal, then create a new consumer group. Please let me know if it's not clear :)
@@DataEngineeringMinds yes this is clear however i am having difficulty in setting up the SSL, i am trying to setup a zookeeper cluster and a broker cluster, i have set it up without any security and the connection seems to be smooth, but when i implemented the steps in ur part 4 video, i am getting errors... Pls can you assist me of you are not too busy? 08181853685 we can chat on WhatsApp for quicker support... Thanks, im in standby now
Do i need to copy the public cert file (ca-cert) across all server nodes? Among the error log, i see something saying "will not attempt to authenticate using SASL(unknown error)" amongst other things....
@@DataEngineeringMinds thank you for ur response, so on the 4 server cluster (2 zookeepers 2 brokers) i have a ca-cert file , should i copy the ca-cert across all 4 machines(ie server 1 ca-cert file should be copied across server 2,3,4 and server 2 ca-cert file should be copied across server 1,3,4 and so on...)?
Yes. It is still valid if You want to use Kafka with Zookeeper. Tha major change as of 2023/24 is removal of Zookeeper (we call it Kraft). I also uploaded a video for that. So, if u watch this one followed by that video, u will get a proper undetrstanding of the latest version
After watching couple of video came here, the flow and way of explanation is excellent, can't be better then this.
Thanks for your support 🙏 Please subscribe to the channel 😊
You are rocking Vinod!
Thanks for your support 🙏 Please spread this channel with your network 😊🙏
Great video Vinod! Waiting for the second part. Keep it up!
Thanks Kevin. I will upload the next video this week :)
this is good, keep it going. Looking for more data engineering playlists
Thanks for the support; Yes, will start doing regular videos in 2 weeks.
how come this video has only 48 comments, man this was really helpful , thanks alot
Excellent explanation..
Thanks for your support 🙏
Excellent video & content . . I have just finished the first video . . I am eager go through subsequent ones. . .
awesome explanation great efforts 👌
Thanks for your support :)
Many Thanks for your effort , Great explanation , I wish you good luck
Excellent explanation...Keep going and all the best
Thanks for your support :)
Excellent bro!
Bro. You are awesome..thanks
great content and nicely explained thank you sir.
Very good explanation vinod. Really helpful for the guys who want to learn kafka. Please share real time scenarios/issues related to kafka. Good job!!!
Thanks Chowdari. I will share soon.
It was very nice explanation ❤️. It will helpful if you share this slides/ content in docs
Well explained mitrama, is there any change from 2020 to 2022
I have a question, if kafka assigns each partition to a consumer in a consumer group, and each partition hold unique messages of a topic. What happens if a consumer 1 needs messages from a topic it has subscribed to, and that message resides in a partition that is assigned to another consumer 2 instead?
All consumers belonging to each consumer group should work towards a particular goal.
Imagine that a twitter account has 4 tweeted 4 tweets and u want to insert these 4 tweets into a database. These 4 tweets are stored in a topic of 2 partitions. Each partition has 2 tweets. Now, if u have one consumer in ur consumer group, this one consumer will consume all the 4 tweets sequentially from each partition and then inserts all the 4 tweets into database; In case, u have two consumers, each consumer will consumer 2 tweets from each paritition in parallel and insert them into DB. In both the cases, the end goal of inserting 4 tweets to database is achieved.
This is how consumers in a consumer group work. They split the work towards achieving a common goal. In your case, as long as consumer 2 consumes the message, consumer 1 should not care because both consumer 1 and consumer 2 should work towards a common goal. In case, you have a different goal, then create a new consumer group.
Please let me know if it's not clear :)
@@DataEngineeringMinds yes this is clear however i am having difficulty in setting up the SSL, i am trying to setup a zookeeper cluster and a broker cluster, i have set it up without any security and the connection seems to be smooth, but when i implemented the steps in ur part 4 video, i am getting errors... Pls can you assist me of you are not too busy? 08181853685 we can chat on WhatsApp for quicker support... Thanks, im in standby now
Do i need to copy the public cert file (ca-cert) across all server nodes? Among the error log, i see something saying "will not attempt to authenticate using SASL(unknown error)" amongst other things....
@@noaexel Sorry... I was busy... The truststore containing the ca-cert should be on all server nodes !!
@@DataEngineeringMinds thank you for ur response, so on the 4 server cluster (2 zookeepers 2 brokers) i have a ca-cert file , should i copy the ca-cert across all 4 machines(ie server 1 ca-cert file should be copied across server 2,3,4 and server 2 ca-cert file should be copied across server 1,3,4 and so on...)?
Thank you ❤
Hi, thank you for the tutorial. Can you share the PPT? Thanks
Super super ❤ you .
Sir, please add subtitle for this playlist, thank you very much
Thanks for watching 🙏 unfortunately, am not able to add subtitles to old videos. However, all recent videos are uploaded with subtitles 😊
Hi Vinod ,
could you please share ppts?
@@tusharpatil7028 hi, have you get the PPT?
is this series still valid for 2024 bcoz its already been 3years that u have uploaded this
Yes. It is still valid if You want to use Kafka with Zookeeper.
Tha major change as of 2023/24 is removal of Zookeeper (we call it Kraft). I also uploaded a video for that. So, if u watch this one followed by that video, u will get a proper undetrstanding of the latest version
@@DataEngineeringMinds thnx for ur reply and what are the new things comes in apache kafka which u didnt teach in ur sessions
Need english subtitles
Turn on the translated subtitles on you