Thanks for really going over the differences between Kafka versus TMQs - Traditional Message Queues - in this video. It helps to understand Kafka for its benefits with retention, replay, and fanout delivery pattern to multiple consumer nodes/services. Also I like your presentation style and the use of multiple colors to highlight the many components and network passing going.
Bro... This is like OS... Semaphoring with producer consumer... There also you send process PCB into queue and when queue is full producer sleeps and when empty reciever sleeps
4:08 Wow the page cache of the operating system need another video on its own. How does it caches, what strategy it uses? What happens if the consumer becomes available and Kafka need to re-read all previous messages? Do OS cache stores anough of messages, should or even can it be twicked?
Yes--tons of OS level enhancements can be made to optimize for specific usage patterns. This is an advanced topic, but we're working on expanding our coverage of these sorts of things in our course at interviewpen.com . Thanks for watching!
3:01 What just happened? Did the offset below(which is the offset for Consumer below) just went directly through 1 and 2? Or was it going through 1 at the same time as the Offset above was going through 1? I mean the offsets and consumers work in parallel, right?
The second consumer hadn't pulled data from the queue yet, so its offset was still at 0. Once it did it received the two events that were sitting there
Hi, I still don't get why Kafka can get much higher throughput than to traditional MQ, is it because of its scalability? If so, why isn't traditional MQ more scalable?
Kafka efficiently uses optimizations built into the operating system, such as sequential disk access and the page cache. For reference, this is covered at 3:25 in the video--thanks for watching!
It's really more about messages being removed from the queue once consumed--this results in different consumer semantics but makes it harder to optimize for throughput at an OS level. Hope that helps!
helpful but shallow. there's gotta be more details. what if the machines running the queues crash? What if publisher publishes faster than consumer consumes? what protocol is used to communicate? if multiple producers are producing, what orders are enforced?
Thanks for really going over the differences between Kafka versus TMQs - Traditional Message Queues - in this video. It helps to understand Kafka for its benefits with retention, replay, and fanout delivery pattern to multiple consumer nodes/services.
Also I like your presentation style and the use of multiple colors to highlight the many components and network passing going.
Thank you!
What I like most about your videos is that they are short, crisp, to the point, great visuals and no BS
Thank you, glad you like them!
Bro you are doing a great job, keep it up💪🏻
Thanks!
Thank you for a nice details explanation @ mechanisms of Kafka
Thanks for watching!
I appreciate you, awesome channel!
Thanks!
Bro... This is like OS... Semaphoring with producer consumer... There also you send process PCB into queue and when queue is full producer sleeps and when empty reciever sleeps
The content is just too good!
Thank you!
Another great video. A great follow up might be Samza. [edit: Samza as a example of stream processing between queues]
We'll add it to the backlog! Thanks for watching.
4:08 Wow the page cache of the operating system need another video on its own. How does it caches, what strategy it uses? What happens if the consumer becomes available and Kafka need to re-read all previous messages? Do OS cache stores anough of messages, should or even can it be twicked?
Yes--tons of OS level enhancements can be made to optimize for specific usage patterns. This is an advanced topic, but we're working on expanding our coverage of these sorts of things in our course at interviewpen.com . Thanks for watching!
3:01 What just happened? Did the offset below(which is the offset for Consumer below) just went directly through 1 and 2? Or was it going through 1 at the same time as the Offset above was going through 1? I mean the offsets and consumers work in parallel, right?
The second consumer hadn't pulled data from the queue yet, so its offset was still at 0. Once it did it received the two events that were sitting there
Messages can be received by consumers in batch, so when the second consumer polled, it received both messages.
Wow.. now I get why it so much more scalable .
How do you determine size of queue, and will you cover usages of multiple queues in kafka?
We have a full video about using queues in large scale systems on interviewpen.com :)
RabbitMQ also provides exchanges and fanout, right? What's the big difference between them in that case?
Yep--Rabbit would be classified as a traditional MQ involving exchanges and messages delivered to one consumer.
Hi, I still don't get why Kafka can get much higher throughput than to traditional MQ, is it because of its scalability? If so, why isn't traditional MQ more scalable?
Kafka efficiently uses optimizations built into the operating system, such as sequential disk access and the page cache. For reference, this is covered at 3:25 in the video--thanks for watching!
So because we have the event dequeueing in traditional MQ we are losing the throughput capabilities?
It's really more about messages being removed from the queue once consumed--this results in different consumer semantics but makes it harder to optimize for throughput at an OS level. Hope that helps!
thank you very much !
Of course!
Can you name some of these traditional message queues?
Sure! The most common option is RabbitMQ, but there are many others. Thanks for watching!
please an review of account system
Is this what you're looking for? ua-cam.com/video/uj_4vxm9u90/v-deo.htmlsi=eaD-ZVsmh2Ha2q3N
But wouldn't saving all msg waste memory how does it handle the it
Default time is 7 days, after which the message gets deleted. You can find them in the kafka-config file
Kafka will automatically delete old messages when it runs out of space :)
helpful but shallow. there's gotta be more details. what if the machines running the queues crash? What if publisher publishes faster than consumer consumes? what protocol is used to communicate? if multiple producers are producing, what orders are enforced?
If you want an in-depth video on how queues are used in fault-tolerant systems, check out interviewpen.com :)
How old are you? :O
You misrepresented "traditional" MQs.
Could you elaborate?