What makes Kafka special? | System Design
Вставка
- Опубліковано 6 лип 2024
- Visit Our Website: interviewpen.com/?...
Join Our Discord (24/7 help): / discord
Like & Subscribe: / @interviewpen
You've probably heard about Kafka, but you really know why everyone is so excited about it? This video dives into the low-level details of what makes Kafka fundamentally different from other message queue systems, and when you should use it over other solutions.
If you liked this video and want to learn more about fundamental systems and how they can be used to solve problems, check out our full course on interviewpen.com/?... !
Table of Contents:
0:00 - Introduction
0:13 - Message Queues
1:32 - Message Routing
2:03 - Kafka Behavior
2:37 - Consumer Offsets
3:25 - Sequential Disk Access
4:51 - Use Cases
6:02 - interviewpen.com
Socials:
Twitter: / interviewpen
LinkedIn: / interviewpen
Website: interviewpen.com/?...
What I like most about your videos is that they are short, crisp, to the point, great visuals and no BS
Thank you, glad you like them!
Thanks for really going over the differences between Kafka versus TMQs - Traditional Message Queues - in this video. It helps to understand Kafka for its benefits with retention, replay, and fanout delivery pattern to multiple consumer nodes/services.
Also I like your presentation style and the use of multiple colors to highlight the many components and network passing going.
Thank you!
Bro... This is like OS... Semaphoring with producer consumer... There also you send process PCB into queue and when queue is full producer sleeps and when empty reciever sleeps
I appreciate you, awesome channel!
Thanks!
Bro you are doing a great job, keep it up💪🏻
Thanks!
The content is just too good!
Thank you!
thank you very much !
Of course!
Another great video. A great follow up might be Samza. [edit: Samza as a example of stream processing between queues]
We'll add it to the backlog! Thanks for watching.
Wow.. now I get why it so much more scalable .
How do you determine size of queue, and will you cover usages of multiple queues in kafka?
We have a full video about using queues in large scale systems on interviewpen.com :)
RabbitMQ also provides exchanges and fanout, right? What's the big difference between them in that case?
Yep--Rabbit would be classified as a traditional MQ involving exchanges and messages delivered to one consumer.
4:08 Wow the page cache of the operating system need another video on its own. How does it caches, what strategy it uses? What happens if the consumer becomes available and Kafka need to re-read all previous messages? Do OS cache stores anough of messages, should or even can it be twicked?
Yes--tons of OS level enhancements can be made to optimize for specific usage patterns. This is an advanced topic, but we're working on expanding our coverage of these sorts of things in our course at interviewpen.com . Thanks for watching!
3:01 What just happened? Did the offset below(which is the offset for Consumer below) just went directly through 1 and 2? Or was it going through 1 at the same time as the Offset above was going through 1? I mean the offsets and consumers work in parallel, right?
The second consumer hadn't pulled data from the queue yet, so its offset was still at 0. Once it did it received the two events that were sitting there
Messages can be received by consumers in batch, so when the second consumer polled, it received both messages.
So because we have the event dequeueing in traditional MQ we are losing the throughput capabilities?
It's really more about messages being removed from the queue once consumed--this results in different consumer semantics but makes it harder to optimize for throughput at an OS level. Hope that helps!
Hi, I still don't get why Kafka can get much higher throughput than to traditional MQ, is it because of its scalability? If so, why isn't traditional MQ more scalable?
Kafka efficiently uses optimizations built into the operating system, such as sequential disk access and the page cache. For reference, this is covered at 3:25 in the video--thanks for watching!
please an review of account system
Is this what you're looking for? ua-cam.com/video/uj_4vxm9u90/v-deo.htmlsi=eaD-ZVsmh2Ha2q3N
But wouldn't saving all msg waste memory how does it handle the it
Default time is 7 days, after which the message gets deleted. You can find them in the kafka-config file
Kafka will automatically delete old messages when it runs out of space :)
helpful but shallow. there's gotta be more details. what if the machines running the queues crash? What if publisher publishes faster than consumer consumes? what protocol is used to communicate? if multiple producers are producing, what orders are enforced?
If you want an in-depth video on how queues are used in fault-tolerant systems, check out interviewpen.com :)
How old are you? :O
You misrepresented "traditional" MQs.
Could you elaborate?