Відео

Thank you for the video! Isn't this exactly what Lambda architecture is and not a "hybrid between lambda and kappa"?

We can definitely go deep on the CDC part, we have normalized data in our main db, but ES is denormalized, how do we handle fanout, shall the CDC consumer call back etc

What is this "Premium AI System Design Mocks" on your website?

Great content. Really liking it. Can you please do a deep dive on TiDB, Rocks DB and Cassandra too

Thank you for this detailed video. But a quick question here - I'm still confused when to use RabbitMQ and when to use Kafka? Because both of them can be helpful for all the use cases

re: the ticketmaster example. Partitioning a single event across multiple partitions would break fairness no?

amazing discussions and pointers as always.. Evan... always look forward to your videos..

How is delayed kafka solution different from the aggregation windows? If we keep the kafka queue alive for 7 days to enable delayed reads internally it is the same thing, only divided not by minute, but by each kafka message. Moreover, there is kafka message overhead. Large memory usage is stated as the main problem of the first approach, but how is it solved with kafka?

Thank you for making such videos, it helps a lot Have you thought about making postgres sharding based on location? I think Uber doesn't support all cities, just a predefined list of locations

Thanks for the video, Stefan.

Good video. Thanks for making it.

Great explanation as always! I keep watching your videos over and over. As a previous commenter mentioned, it would be great a deep dive on ZooKeeper, it’s mentioned many times in Orchestration/Coordination scenarios along with Consistent Hashing and I think it would be valuable to understand how it works.

In terms of Horizontal Scaling, from the accompanying article: "Horizontal Scaling With More Brokers: The simplest way to scale Kafka is by adding more brokers to the cluster. This helps distribute the load and offers greater fault tolerance. Each broker can handle a portion of the traffic, increasing the overall capacity of the system. It's really important that when adding brokers you ensure that your topics have sufficient partitions to take advantage of the additional brokers. More partitions allow more parallelism and better load distribution. If you are under partitioned, you won't be able to take advantage of these newly added brokers." My understanding is that Kafka can be scaled horizontally dynamically, perhaps as a system sees an unanticipated increase in volume. If thats correct, does the above imply that partitions can be added dynamically too? In the example cited, LeBron James campaign, I took that to mean that you'd add extra partitions for that campaign in anticipation of the additional traffic. In the case of hot partitions, can one of the prescribed techniques ( say random salting or compound keys ) be added on the fly? If this is non trivial can you maybe link to how this is achieved? Thanks so much!

I have a system design tomorrow and this is a perfect timing to watch!

You have amazing teaching skills! The World Cup example was incredibly good and entertaining to watch. I’ve paused my interview journey for now, but I always watch your videos for the pure knowledge they provide. Thank you so much!

Thank you. Love every video published so far.

First off, thank you for these videos and resources, they are very valuable to anyone studying for interviews. I'm curious though, how would you improve the interview process as someone who's been on both sides of it for a number of years? I question the value of these interviews given that people are being asked to design massive systems, for billions of users, engineered by hundreds/thousands of people over a number of years, which were iteratively improved over time. They're expected to have a pretty ideal solution by having researched the problem or similar ones ahead of time, or much less often, having faced similar problems themselves. If someone was asked to actually design a system in a real production environment, they would spend ample time researching it ahead of time anyway, so I don't necessarily understand the value of them knowing it up front in an interview. I'm also curious how you would react if you were still interviewing people, and a candidate proposed a solution that's an identical or near-identical copy of yours. Would you pass them as long as they understood why each component is needed, and why certain technologies should be used over others? Would you have time to properly gauge that in a 45 minute interview once they've finished their design?

Thank you for all you guys do!

@hello_interview I'm confused about the way the distributed lock works. User 1 calls reserve and acquires the lock for a ticket to Eminem's concert. User 2 calls reserve but the system responds saying that ticket has now been reserved User 2 is a malicious user and calls confirm_ticket before User 1 calls it How does the reservation system in this design prevent User 2 from stealing the ticket from User 1?

I don't know if you'll see this, but please read if you do. I've been having a lot of trouble lately, and I would really value even just some very brief advice. I've been watching your videos a lot to prepare for an upcoming interview. I'm a PhD computer scientist, but I left professorship about a year ago to work in industry. I've been REALLY unsuccessful in landing anything, but there's never any feedback, so I don't know what I'm doing wrong. Just a lot of anonymous rejections. I'm literally facing bankruptcy. I have an interview on Wednesday, and so I'm learning all I can about modern system design. I'm hopeful, but trying not to get too excited. I'm really thankful for your videos and all the information you provide, though. It's really helping. Hypothetical question: if you became a single parent with no education at 24, but buckled down and raised your kid and took care of your sick mom and made it through an undergrad and a phd over the course of 10 years, and maybe don't have a huge professional footprint because your grad work + all the course materials you made don't add up to a very impressive github profile, what can you do to stand out more? Keeping a house running and raising a kid and taking care of a sick parent doesn't get many stars on your repos, but I think it speaks a lot to adaptability, perseverance, problem solving acumen, etc. But those aren't things you can typically bring up in a cover letter or resume, and you're not supposed to talk about it in interviews, so it just looks like I haven't done anything serious outside of my grad software. It feels like a catch 22. Can you give me any advice? I could wallpaper a house with all the rejects/ignores I've gotten. Thank you again for your great videos!

I always struggled to choose between queues, stream and pub-sub but this video makes it super easy to understand what to use and when to use.

Thank you so much. Love your channel. Please provide a deep dive on Redis too. 🙏

Superb. Someone who has just theoretical knowledge on Kafka helped me understand this "topic" little better. Request for a video on ZooKeeper (I think Kafka moved away from ZooKeeper to kRaft)

In the section about using Kafka for messenger, how would the topics and partitions for a messaging application like Messenger be structured to achieve low latency and high throughput? For example, if there are 1 billion users on the platform, would there be one billion topics, or a single topic with a billion partitions, one for each user (which I don't think is possible since the recommendation is 4k partitions per broker and max of 200K per cluster)? Is there a different approach that could be considered? What are the tradeoffs for each option? And great video. Thank you for doing this.

Listening via AUX while I’m driving. Love it. Curious to see it visually later.

Love these deep dives, thanks!

Good going, please keep continuing this series! I had a question regarding consumer concurrency, which is not discussed in this video. Let's say I have 1 consumer group with 2 consumers running and topic having partition of 8, then each consumer will be assigned with 4 partitions when concurrency = 1, how the consumer gets affected if consumer concurrency is changed to 2 now ?

For handling kafka consumer down we could turn manual commit offset on, There's option of AutoCommit Offset and a timer limit also when we should autocommit. Though Great video for revision for kafka. Also it would have been great if you had mentioned number of kafka consumer application limitations based on number of partitions.

Liked even though I haven't watched the video. I know it will be a banger !

You are the best at what you do, after listening to your videos I cannot now like and tolerate other system design videos, please make more content for system design

Thanks so much for these deep dive videos 🙌. Also your system design videos are very helpful in my learning journey.

Most awaited topic. Thank you for the detailed and insightful video on Kafka. Your every video is a gold mine. 🙏🏻 ❤

First! Shoutout to this channel! It really prepared me for all my system design interviews this cycle

Very nice explanation! When actually crawling the pages, it could be blocked by the website owner. Do you think we need to mention this in the interview and provide some solutions like using rotating proxies?

The best system design video i have come across, main thing i did not realize 1 hour went by

Amazing content

There is no mention of shardimg here ?

Step one: use weak authentication for your organizational logins.

The flow is top notch.. Great videos guys 🙌

is this asked in product design or system design at meta

If we increment the video freq and then insert that videoID:freq pair into the heap if the freq is bigger than the floor, isn't it possible that the heap would have multiple entries of frequencies for the same video? For example, say k = 3 (So I assume heap size is capped to 3 in this case): watch event for video1 comes in, so counts = {video1: 1}, heap = [(1, video1)] watch event for video2 comes in , so counts = {video1: 1, video2: 1} heap = [(1, video1), (1, video2)] watch event for video1 comes in again, so counts = {video1: 2, video2: 1}, heap = [(1, video1), (1, video2), (2, video1)] In order to test if the incoming element is bigger than the floor, we have to use a min heap so that is why 1,video1 is at the top Say another watch event for video1 comes in again, then: counts = {video1: 3, video2: 1}, At this point we could remove (1,video1) from the top of the heap (since heap size == 3 now) and get: [ (1, video2), (2, video1), (3, video1)] Basically the issue is cause the frequencies of the video changes over time, there could be old freq data and new freq data in the heap at the same time for the same video. Because of this, I don't think just heapifying the existing heap is enough. I think what we would have to do is run through the entire counts to generate the heap. This is much costlier than heapifying though. Heapifying is O(K), while the former is NlogK, where N = # of videos on each shard, and N could be pretty large.

The sound at 50:25 is more than critical to pass the interview😅😅😅 where do I learn that?

I had a couple senior engineers I worked with early in my career who ripped apart my PRs and my designs, which helped me learn a ton. At some point after sitting through a bunch of design reviews (most of which I didn't author) and listening to the senior/team lead ask all these questions, suggest we stick a queue instead of something and make it async, talk about fan-out strategies, the simplicity of immutable data vs. mutable, etc. I really learned a ton and took those lessons with me. I still had to study a lot for design interviews because I never actually worked with web sockets and so on. I can't imagine that many people, even at FAANG companies, have created systems from the ground up (instead of coming in and building on top of what exists) that serviced millions or hundreds of millions of users. My first couple rounds of design interviews I messed up for sure.

400gbps nic😂

One question I have with this problem and your solution is, the top k heap will constantly need to be updated with views coming in for the videos inside. i.e. lets say you have 3 video with views as [v1: 100, v2: 150, v3:200], if we get chunk of views for v1 (say 75), new heap will look like [v2:150, v1:175, v3:200]. How would the heap handle modifying values of videos that exist within the heap. Another problem with a heap is, read entire heap is usually in the order of nlogn and would require you to poll entire heap(unless there is some other method I am not aware of).