System Design: Why is Kafka fast?
Вставка
- Опубліковано 29 вер 2024
- Weekly system design newsletter: bit.ly/3tfAlYD
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
Other things we made:
Digital version of System Design Interview books: bit.ly/3mlDSk9
Twitter: bit.ly/3HqEz5G
LinkedIn: bit.ly/39h22JK
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.
Subscribe and Kafka will say thank you :)
ok, it's done Sir
What software do you use to create this awesome motion graphics?
May I know what tool you guys use to make these animated videos? Just curious..!!
I just discovered this video in my feed. _Sometimes_ the UA-cam algorithm actually works! 🤠Great video! I just subscribed to your channel!
I wish you were my professor in college.
The absence of any background music makes this video great.
This comment. Yes.
fully agree!
amen!
i agree
Exactly
Mentioned a lot in the comments, but I have to say as well: what a great explanation, straight to the point, no bs and gives enough info without overwhelming with details. Thank you!
Wow, this one is super cool. No background music, cool minimalistic diagrams, calm voice!
Seriously, thanks a lot Alex for all the stuff you convey through your LinkedIn network and UA-cam videos. Just love the way you distil the topics and make them understand beautifully.
I love all the System-design Content posted by you!
Thanks for sharing your knowledge! 🙏
You guys are doing amazing work here. I love the aesthetics, pace, explanations, topics, and cadence of it all. Kudos!
Essential collection of videos in this channel for a software developer
Wow. Never heard about Kafka, but after this brilliant video now I know why it is so fast. Still no idea what it is, though. And so many totally not astroturfed comments. Sweet.
i wanted to comment that i appreciate the level of detail in the explanations in the video.
looking forward to more useful content!
Very clear explanation. Thank You!
Awesome Explanation about Kafka is amazing...Thank you, Alex
Very simple with good animation to explain things clearly. Keep publishing these kinds of useful videos.
Loved the animation and explanation. Keep enlightening us all!
We need so much more of this.
This is an amazing video.
Actually putting it out there - I LIKED AND SUBBED!
Well deserved for great content 💯
Great video, only one more thing could be mentioned: send file doesn’t work, when TLS is terminated on Kafka.
wow the comments are right. simple and clear... subscribed
Nice intro about Kafka, learned quickly, now you got a new subscriber 👍
Me, who has zero interaction with Kafka:
"Hmm yes, very interesting, liked and subscribed."
very nicely explained 😊
How are such animations made ?
Short & sweet. Thank you.
nice simple explanation
beautiful footage ❤
Very elegantly done. I wonder what animation tool they are using ??
Awesome video!!!!!! How those animations are made? In after effects??
I was hoping for an explanation about how to read is efficient when Kafka only performs sequential write...without actually knowing Kafka, I'd guess sequential page tables?
I feel like I'm getting some shaolin training just listening to this man
Wow this was incredible
The tutorial is useful. Thanks for your sharing. Could you give more explanations on how Kafka enforces sequentiality characteristic on the disk? Do we need a specialized disks or dedicated settings because as i know a file can be stored fragmently on the disk?
Useful information.
Can you bring more about kafka?
Thank you for great explanation and putting so much effort
@ByteByteGo what exactly is the use of socket buffer , is there any consequences if we dont copy the data from disc to application buffer and Socket Buffer ?
it's just phenomenal elaboration!!! is there any way to get such thing over book or yo have any course available in any online training organization???waiting for your quick immediate response
Thank you so much
Amazing video !
I don't understand the hard disk section. Is part of kafka's selling point that you can use it on HDDs?
Yes. Kafka came to life in 2011 and SSDs were just getting started then. But even now SSDs did not replace HDDs in clouds and data centers. You can see that new, larger HDDs are still being introduced to the market. If you use HDD then with Kafka you can technically never remove your messages since storage is cheap and performance doesn't take a hit. If you keep your messages then you can replay them from any moment which has its benefits.
Love your videos. Can you please share what software do you use to make the graphics ? Thanks
Ahh, great work!
May I ask are you the author of the book of system design?
I'm bored, it's 4 AM, and I was waiting for a tldr of what Kafka is, ngl.
Excellent video. Are there any situations when Kafka cannot use zero copy and has to resort to read flow without zero copy? Thanks
When doing ssl based connection to brokers - so any real world situation where you store any form of PII and don't want it potentially "seen" in transit. But with cheap servers... You generally don't care, it is extremely fast anyway, as sequential read/write and most consumers being on the head of the partitions, and cached in memory by os, the backing topic never gets read (The interesting optimisations are the time index and record offset files to allow pseudo random access / positioning in partition files)
awesome!! 😃
Exemplary Illustration on Kafka
Where did you get the numbers for writes per second from?
awesome
can we compare these 2 concepts with traditional queue RabbitMQ/AWS SQS?
If your systems are appending records to the end of lists constantly without reordering them, won't it be problematic for the client application?
Wait, but how does encryption work with zero-copy DMA?!
Is there difference between the 2 volumes of the book?
me not having a clue what kafka is about
Simple video without being a pretentious tech bro.
What an amazing tutorial: Just the necessities, no annoying background music, no annoying calls to "subscribe and like".
If all youtube channels were like that, we could heal the world.
Also, I checked your channel page and was shocked to find that this was only your 3rd video.
Keep being awesome!
This Tutorial is insanly "Zen" but he said "please subscribe" right at the end :P
These videos are amazingly simple and clear. The animations are spot on!! Too good xD I wish this channel never stops uploading new content
I 100% believe you should make a whole series on Kafka, your way of simplifying the subject is legendary.
The index to the shards on the Businesses DB should be user_id or business_id? I think the latter makes more sense not sure if it was a typo on the video
The bigger the diamter of the pipe, the larger the amount of liquid that can go through it. Put some nsfw tag on this thank you.
wtf? that whole video is about distinction between sequential and random access on... magnetic based storages, which are now used as a last step/level in caching sequence... (quite like magnetic tape storage back in the days...)
this "Advertisement" for kafka is speaking about hdd vs ssd (distinction which was valid few years ago) ignoring that there are things like nvme which are in the middle or even closer in cost to ssd, being only ~10x slower than ddr kind of memories (while ssds are ~100x slower than ddrs)
nicely done video, but srsly strongly misleading...
How can one keep things so deep and yet stunningly simple. Hats off!
Great video!
BTW, how to create such kind of animations?
Short, high quality, clean and extremely precise content...Many Thanks!
is there a risk of Kafka accessing other areas of the memory cache in which system calls could send wrong/private data?
Why is it called zero copy when it clearly shows that there is one copy made to the NIC buffer?
This is a great question. Let's use Wikipedia's definition of zero copy, which says:
"Zero-copy" describes computer operations in which the CPU does not perform the task of copying data from one memory area to another or in which unnecessary data copies are avoided.
The "spirit" of zero-copy is to eliminate unnecessary data copies. In Kafka's case, sendfile() does reduce the number of copying. In the best case, if a NIC which supports scatter/gather DMA is used, the copy does not involve the CPU, making it truly zero-copy.
There is quite a bit of nuance when it comes to zero-copy. Maybe we can make a video about it down the road.
Thanks for the explanation Alex
amazing video i love the animation so much , what application do you use to make such animation?
影片中說明兩個為什麼 Apache Kafka 能夠提供高流量傳輸大量紀錄的特性：
1. 循序 I/O
以 C 來說，當使用 fopen() 需要開啟一個檔案為 append 模式，file pointer 會直接在檔案尾端準備以新增方式繼續加入新資料，會比每次加入資料需要移動 Pointer 到特定位置再寫入來的快速。如果用硬碟的循序讀寫與隨機讀寫，會更容易理解。
在 File-based Database，例如 dBASE, COBOL + ISAM, Paradox，也是直接將新紀錄寫在檔案後方。可以用 PC-Tools 打開檔案觀察 HEX Code 確認。風險在於如果來不及寫入 EOL，沒有順利關閉檔案，就會造成檔案損毀與資料遺失。
刪除紀錄也只是在記錄上做個標記，並不會真正刪除，需要等到執行 compact database 才會真正刪除。因此我在設計需要確實刪除客戶個人資料時，會以無意義的字串覆蓋，直接刪除其實只是標記，資料還在。
2. [Zero Copy](en.wikipedia.org/wiki/Zero-copy) 避開將相同資料在不同記憶體區塊再次複製後移動，縮短傳送路徑。例如在提供 DMA 模式情況下，讓系統函數直接將讀取已經被讀入記憶體緩衝區的資料放入網卡 NIC 緩衝區開始傳送，省略 Socket Buffer 路徑。
Uhh I thought this was a final fantasy video
Hi Alex, Just a suggestion, please make some videos on different consensus algorithm (RAFT, PAXOS).
Disk read and network transfer delays are order of magnitude longer than copying between buffers
This is not the same Kafka I was expecting, but happy to learn. thanks for sharing!
I don't know a single person who has a Kafka scale problem and not enough money for SSDs. And essentially every modern DBMS will use a WAL and indexes in sequential space too. And the speed comparison is hard extremes in the range of grabbing single bytes off a 4k read. You add more than one consumer (or, dare I say it, write at the same time that you read from a partition) and your perfect scenario where you never seek goes poof. Please put Kafka on SSDs.
Asking why Kafka is fast is really the wrong question. Under those two conditions, SQLite is fast. It even skips the network! Why don't you ask why Kafka can scale horizontally without much effort, for instance?
And please, be measured in your chosen technology. Kafka is a big iron and not likely to be useful in a lot of architectures. Yes, you can use it as a message broker for your queue, but that's really not a good use. Add one partition and now your FIFO is guarantee is out the window. Kafka is for sharing data and continuous updates on the same with an unknown, but usually large quantity of other services. And usually in an enterprise.
Oh and also, please don't tell people the latency hierarchy is a lie. Access patterns improve latency, but memory will always be faster than any disk.
Each kafka broker could have data from multiple topic partitions. Reading data from same kafka broker still needs to swich disk header. Will that hurt the benefit from sequential IO?
Never heard of Kafka. Thank you UA-cam algorithm.
What are you gonna do with this knowledge now?
@ByteByteGo - How do you create those animations? Which software, what process?
In 5 minutes I learned a lot! Amazing video!
You are a good teacher!
Thank you and I hope to see more videos from you!
3 videos in 14 days and 48k subs ,nice
Cool stuff, the concepts described here are known by embedded system designers who suffered enough.
So the question is, why before it had to first copy into socket buffer, and not anymore with Kafka?
ASMR experience :)
i have subscribed...
Neat explanations...
I am not curious about Kafka, but curious about the optimization techniques and strategies they have accomplished which I would like to learn...
Please do more!
This video explained nothing. Just a bunch of buzzwords.
When using DMA sendfile(), is it transparent to the receiver? I mean does the receiving side have to do anything special or just receive the file through the normal socket as usual?
Amazing details about frequently used software. Lucky to bump into this page. Thanks
So this video was in my recommends so I clicked it. 1/2 way done and I still don't know what kafka is. Is it a database? Is it a SAN? Is it a program that runs in an os to transfer files? Wtf is kafka! So I got to the end and I still have no clue what kafka is.
Same! 😂 Clicked on recommendation, still idk what is Kafka.
Moreover, I studied about such optimizations in college back in 2000s, as example if anyone opened file without memory-mapping, it got low grade. WTF, why they told about such basics which use any ordinary developer?
This helps to explain why the sequential read speed of HDDs is on the AWS Cloud Solutions Architect study guides.
Nice work!, I would like to learn the content of your book in the video format.
I have used kafka before but never had to think about why it is actually fast. This was very informative. I like the format of the video as well
What software are you using to make this awesome presentations?
Very informative video, thank you so much!!
I had only thought about sequential IO because of the append only log file.. this was insightful
are there any security implications of zeroCopy? Since the data is directly being written within the kernel space how does it honor the boundary?
Yes, you can’t use TLS. Transparent memory encryption (e.g AMD SME) may also prevent zero copy, and you probably want that enabled on a server.
excited to see Sahn on youtube!
this is by far the best tech video I've watched. concise without losing any depth! looking forward to more videos like this.
I've had the fortune to (indirectly) work with Sahn and review his code. one of the few top talents that any company is lucky to have. this video is as high quality as other production of his.
2 questions for Sahn:
1. there's a small disconnection between "sequential IO throughput vs random IO throughput" and "HDD vs SSD". is there any perf number difference on sequential IO throughput on HDD vs SSD?
2. is there any perf number difference(ops per sec or latency) for zero-copy vs traditional buffer copies?
Great technical explanation. I just want to add that Kafka can be used for much more than just data ingestion sending data from a data source to a data sink. The Apache Kafka open source project also includes Kafka Connect for data integration and Kafka Streams for data processing. Therefore, you can leverage the characteristics explained in this video to build a modern data flow with a single (scalable and reliable) real-time infrastructure instead of combining several different components (like Apache Kafka for ingestion, Apache Camel for data integration, and another stream processing framework like Apache Flink for real-time analytics).
Reliability of Kafka has yet to be proven. Ever so often it does not meet data integration core requirements on reliability, especially in the area of disruption and recovery, where it quickly says GoodBy to “At-most-once” semantics. Don’t get me wrong, Kafka is really great for what it is designed for: efficient streaming in BigData architecture, but that architecture will tolerate a certain fuzziness of data, which pure data integration architecture would not allow for.
your video is very clear and on-point Sir, thanks a lot 👍👍
Very deep insight! Looking forward to your next videos, please keep going
Hey Alex, love your content.
Still waiting for SD Volume 2 in Amazon India 🥲
Great video! May I know what do you use for those animations? These animations are basic yet effective!
Amazing explanation. Thank you sir.
I really appreciate your work. Excellent video. Superbly Articulated. Easy to grab the concepts. Great work. 😍
Really great presentation! I was scared when I saw Kafka but you explained it really well.
Thanks for the useful instruction!