Very well explained. Just one point of discussion here from my end: Can we create some inventory-service and keep availability there instead of booking-service, this will help in well defined boundaries and responsibilities. Booking service should just keep the booking information and when booking is done, it can update the availability in one single distributed transaction. Let me know if there are any challenges/problems in the above said approach.
Excellent video overall. Automatic scalability: How much is the application developer controlling it (scripts, architecture, etc) and how much is the vendor component/cloud service provider (thresholds, metrics, etc) itself doing it? I see a component re-usage pattern. User Service, Hotel/Cab Service, Booking Service, Search Service - Elastic Search, etc across your videos. Great to see the modularity being put into action. You talk about trade offs and justify design choices which is a good but also reinforcing about first principles of components. Much more than an interview question which reflects knowledge and experience. Only disappointing thing is it is rendered by poor voice recording. Sigh!
Great videos on system design Sandeep. Thanks! I have one concern about the Airbnb design. You have used Kafka for multiple things and few of them don't really require or would be actually a not-so-good design. 1. Hotel onboarding: This hotel onboarding wouldn't really require an async service as I don't see this system as write-intensive. A simple sync REST API should work. 2. Booking of a room: The booking of the room by a user & removing the room from the available list should be consistent. I think here also Kafka's use should be avoided. In both cases, immediate reflection to the other users should be handled by a write-through cache policy. Let me know your thoughts. Thank you!
1. Why is Kafka not a good choice? There can be many workflows that we can trigger whenever a hotel is onboarded, like sending a notification to customers, advertisers etc. All of that can be decoupled into a separate service and the service can consume from Kafka
Thanks for great System design series. One doubt w.r.t. DB design here: I assume search API will look like search(lat ,long, startdate, enddate, # of rooms) to return list of hotels in the given lat long area. Perhaps we can use Quad Tree type data structure here if we want to reduce latency or update booking ASAP. (1) Available_rooms table has a field for available_quantity, Will it carry 365 entries for date column for say 1 year? If not what is more efficient way to store this info and perform fast search. Also don't you think PK should be room_id+date above for uniqueness. Please correct me if wrong. and then (2) How are we connecting Available_Rooms with hotel.
@theghostwhowalk - I had exactly the same concern as "(2) How are we connecting Available_Rooms with hotel." - Typically there should be an association of hotel in the 'Booking' DB as well. Else how for a given date range would a list of hotels appear based on room availability. @codekarle - Sandeep can you help out here ?
The content is awesome, really helpful to me. Please make videos on geographic information systems like- Google Earth, Street View, Voice recognition systems like- Siri, Alexa
Great video, thank you! Just a question - 1. Why don't you use Redis for the other services too? 2. Why don't you use Casandra DB for Hotel service too? 3. The arrow from my SQL cluster to archival service is strange to me. How does MySQL send messages? It wouldn't be better that after booking service is done booking then send message \ HTTP request to archived service? 4. The archived service has only one purpose, just to insert data to Cassandra. Isn't it overhead and the logic should be in the booking service? Thank you again :)
1. Why don't you use Redis for the other services too? - Usually I prefer to do this when the read-writes ratio is very high. If there are a lot of writes then there is not too much benefit of caching. Usually these decisions would be data driven based on real access patterns. 2. Why don't you use Casandra DB for Hotel service too? - Cassandra could be used here, but I believe on this kind of dataset, you'll have lot of random queries happening, to fetch hotel by id, to fetch hotels in a region, or to fetch hotels managed by a user, etc. This would be for a lot of internal tools mainly that every company has, which we did not go over int he video, but those tools are always there. Now that query pattern is not optimal for Cassandra and we'll need a lot of data duplication to build those tools, Mysql does a better hob there. 3. The arrow from my SQL cluster to archival service is strange to me. How does MySQL send messages? It wouldn't be better that after booking service is done booking then send message \ HTTP request to archived service? - Those arrows are just to call out reads and writes. I know it's a bit cluttered there, but mysql is not sending any messages. Archival usually should be a bulk batch job, and not done for each booking, and it makes it more optimal, since you can run this at low load times though there is nothing stopping from doing it the way you are suggesting as well. 4. The archived service has only one purpose, just to insert data to Cassandra. Isn't it overhead and the logic should be in the booking service? It could be the same service in a small scale system, but at scale this would cause problems. Think of it in a way that when archival begins, it would probably shoot up to near 100% CPU utilization for a few seconds while it is running unless you are throttling. If thats in the same service, it'll impact other features of booking service. If you have it as a separate service, you can run that out of just two machines/maybe one docker container only, while booking service can scale as per your bookings traffic. Hope that answers your questions :)
Hello I am new to system design and learnt most of what was explained. I still need some help on archival service and cassandra. Why do we need an archival Service?
That is for book-keeping purpose and incase a user wants to see an older booking, which happened let's say 2 years ago. That become is not being edited, so you don't need any ACID compliance there.
@Sandeep : Nice discussion. Apart from concerns(which people have already pointed out), that if we connect booking and new hotel posting with kafka, the system will become near real time for search rather than real time which I see can be improved by using SQL DB only to power the search. Also, Kafka has some retention period like 7 days , so powering elastic search from Kafka rather than the SQL DB will result in inaccuracies in listing the hotel. please let me know your thoughts on that.
I think here the intention to use Elastic search is to support Fuzzy word search and for Word search Elastic Search DB is best as it optimised for that and gives results in best Time complexity. Also, we can do it with SQL DB but the point here is scalability and when more than a 1M concurrent users tries to search at same time SQL will be overloaded. Also, kafka retention period is nothing to do with elastic search it only retains events produced.
Can you prepare the same for movie booking system? It looks similar to this, but in movie booking, user has the freedom to choose which seat to book. The DB schemas might be different.
@codeKarle please make video for 'Design metric publishing system for cloud environments', for 'Design shopping cart' and for 'Design multiplayer game'. Thanks!
Nice Video. I just have a question on inventory. So here we are keeping the hotel rooms inventory in ELK for powering search and also in the MySQL DB (available_room table) for booking service to interact with?
Glad I found your channel - the best System Design videos, well structured, organized and easy to follow. Quick question - what it MySQL master becomes a bottleneck, for example there is a sporting event and a spike of bookings? Read path can be scaled by adding more slaves but how about write path?
Thanks!! One obvious way would be the usual sharding, but that complicates the maintenance process. I would rather split the servers in multiple groups or rings, with the intent that one group maintains the data for a few countries, and in case of planned/predictable spikes, you can throw more hardware in the groups where you expect more traffic for a couple of weeks when the traffic is expected to increase. This ring would not just be on the DB front, but also for the Application since you'll need to scale up the app servers as well most likely. I am assuming here that we know what kind of spikes would be there, but most companies these days have that kind of a Prediction Engine build accounting the social activities/historic data/etc, so that's a fair assumption I believe. Do share the channel with your colleagues, it helps :)
@@codeKarle Thanks Sandeep! By splitting into multiple groups, we are still looking at having 1 master per group right? Otherwise if we add 2 or more masters we are essentially sharding, right? The question is whether can the spike within one country/city cause write traffic throughput increase to the point that a single master becomes a bottleneck and won't be able to keep up?
To optimize writes sharding+ replication of each shard is one way to support writes and reads. Sharding however is tricky on relational Dbs. Another approach would be active active replication across the cluster with leaderless quorum
He is mentioning about redis callbacks to use when we didn't receive payment within specified time. But what if the redis itself is down for sometime ? ( as redis is in-memory key value store ) How are we going to make a callback which will mark the booking status as cancelled and increments the available room count.
@codeKarle Shouldn't we also talk about about scenario where two hotel broker trying to book same room. For eg: AirBnB and MakeMyTrip. What should be source of truth in this case?
Great question. But more than a tech problem, it's a business headache to manage, so all these companies so a slightly different thing. Usually if the hotel has 50 rooms, they would give off 10 to AirBnB, 15 to MMT and so on. Mostly these companies pay for these rooms upfront even if there is no customer there to get those rooms all for themselves. The benefit it gives is that each of these companies say AirBnB now has exclusive 10 rooms that it works with and takes care of booking, and they could play with demand-supply maths and change the pricing to make more money. That being said, we could still build the system for what you are recommending. It's not a big change from what we have here, but the solution highly depends on the tech capabilities of the hotels, and who maintains the master list of inventory for each hotel.
how do deal with concurrency? two users trying to book the same room. first user from 3rd to 8th and second from 6th to 10th. how this situation will be dealt with if they are trying to book at the same time?
@10:10 "thereby making sure that this is having a scale that MySQL can easily handle". Can you explain or write the blog post on hotel booking system design.
For DC replication , what happens if the person connects to the R2 DC (from US) and tries searching for hotels in region which is NOT maintained on the data-store for that region ? The whole data layer needs to be replicated, I think
All the data will be replicated to each data center that was opening statement. Partition data just acts like multi-master. But data is still replicated. Other DC can act as slave, where reads can happen.
Great video. I have one question though. How are you generating/updating the available_rooms table? Where's that data coming from to begin with? Thanks!
Really great video : Have some doubts below - So how are we storing available table. Are we going to store for 365 for each hotel . That will be inefficient. - Which service will pull data from hadoop cluster? once the analysis done from hadoop cluster aren't we storing it in Analytic DB? - Should not we store images first to s3/gcs then later cdn?
Excellent points. I had the same question. I think we shouldn't write images directly to CDN. We will store them in an object storage and CDN will cache it based on usage of the files geographically.
In case anyone reads it now, I believe we are anyways dividing the entire geography into parts in terms of data centers so it won't be exactly a single table populating the data for all the hotels and its room types for 365 days. It will be more like - If I divide the entire geography into 4 parts where each part will have a separate database, so available_rooms table can contain 1 million/4 = 250000 hotels * 10 room categories * 365 days = ~750 million rows. Ofcourse this data will be sharded based on hotel IDs so it will be separated into separate shards.
what if customer booking for hotel and at the same time hotel delist their room. Since the databases are different it would definitely create inconsistency. How are you planning to handle that ? Will you be confirming the room to both the databases ? User generally spend 5-10 minutes to make the payment, and in between if the hotel is delisted then how it is handled ?
@CodeKarlw I like this design, BUT: - Using a RDBMS for performing the booking lock may not be appreciated by all interviewers: I proposed a very similar architecture in a sr backend engineer interview @ Uber and I've been rejected with the motivation "He used a RDBMS for performing the reservation lock". It would be interesting seeing an alternate implementation. (Redis lock?) - The service crash scenario is completely absent in the design, specifically a crash of the booking service after a payment has been successfully performed but before the booking status update - How do you maintain the datacenter database aligned? Shared kafka topic consumed by an align service? Thank you for your work anyway!
I don't understand a few points here - 1. How is the available quantity only a function of room_id. If we are accepting date range in POST /book, it should be accounted for in the schema. 2. Assuming, we have that in the schema, we should also need to have an efficient query that checks for availaibility based on these three criterias.
Superb Sandeep. I am looking for Coupon management system design. its commonly asked across companies. On internet, there is not a single design for this.
Great video Sandeep. Quick tip for Product Managers looking at this - I think the first 15 minutes and then 32:20 onwards is pretty relevant. Wanted your thoughts on something: The safest way to interact between the booking service and the search service would be to have a single DB where locks are taken on the respective fields when a booking is made. This would guarantee that no room is double booked ever. That approach seems very very 90s and theoretical. But is there ever a case in your design that the delay between syncing the search and booking service through Kafka can cause a double booking scenario. Are there any other cons you could think about this architecture? Thanks in advance.
Hello, we are using only one DB for booking management. There will never be a case where there would be a double booking. there can be a case where in search experience hotel is shown as available but on trying to book - it might be shows as unavailable. But this should be in only very minimal cases
Nice Explanation , I am having one doubt here , From Hotel Service Storing the data into one MYSQL Cluster, and from Booking Service Storing the Data into Another MYSQL Cluster, then How do they in sync ? I am mean Hotel Service data and booking service data in DBS.
Thanks for article. Live updates to the Hotel will get published to Kafka and it feeds into Elastic Search. But my question is, Elastic Search only holds live updates or its like a replica of Hotel DB ?
For the issue where TTL expires before the Payment success message, why can we set the TTL long enough (more than Payment service SLA) for us to be absolutely sure the payment will never be successful later. Con is definitely that the room is reserved for more than it should be blocked for, but we definitely reduce some complexity there. Kudos for the effort
Good video. Just one concern, how come room id will have quantity, I think room id represents one room and quantity should always be one. We should have another table which maps room id to booking id.
Thanks for the great explanation. Question: Why do you suggest the approach of setting the boolean "isActive" to false, to delete a record rather than using the SQL delete operation?
It's generally not a good idea to delete something from the Database. If you need that information for auditing, financial reporting, or just debugging, it'll be helpful to have that information in our DB marked inactive. Additionally delete queries so have a risk of deleting something accidentally in case of bugs, so i'll always prefer to have a delete/isActive flag and filter the active records in get queries.
Hi Karle, I have one question. Will Kafka only contains all the hotel/room data, same as a database? If we treat Kafka as a source of truth for the input of search service, and some room data that exist in the database are not in Kafka, the search result will always miss that part of room data. In addition, the Kafka data will be expired and evited in 7 days, which means some room data will not be able to search by end-user to book, even though they are available, just because it does not exist in Kafka, and we treat Kafka as the source of search service. Could you help to clarify it? Thanks.
It isn't Kafka where we keep this data. We will keep this in ElasticSearch. Kafka is just a message broker and passes all the info from sql db to elastic search for better search capabilities.
Great video. One thing i that i found missing in this design is : When admin of hotel adds new rooms -> there is no flows that updates rooms availability in DB of booking service Please let me know if i i'm missing anything on this
Can anyone help me know why SQL Database(MySQL) is used in Hotel Service Application, wouldn't having NoSQL be correct choice. I see a point on using SQL in Booking Service.
For hotels data store images in s3 or any blob storage and metadata in MySQL with ref to s3 as a link and then have cdn pull data from s3 which can be accessed via the user app along with metadata for the hotel. Not sure why cdn is being used as store for images in this design..
For hotels data store images in s3 or any blob storage and metadata in MySQL with ref to s3 as a link and then have cdn pull data from s3 which can be accessed via the user app along with metadata for the hotel. Not sure why cdn is being used as store for images in this design..
the way you have linked CDN to hotel service is wrong. The way CDNs work is they have edge nodes (the one that you have represented) and origin servers. The CDN is always seeded by the origin server when the client requests the data. You don't upload to CDN or make any modifications to the CDN as shown in the diagram.
Contentful video,very nicely explained .. Basically i was looking out for scenario , where multiple people select the same hotel almost at the same time , or like how we do solve the race condition in such a huge system ..? by avoiding same booking with multiple people.
The property ISOLATION of ACID properties deals with that. So, if you set the Isolation level of a transaction (T1) to be Serializable, no other transaction would be able to write into the tables involved in the first transaction (T1). So, whichever transaction starts first, that transaction will end first. In the case of a booking scenario, let's say, T1 and T2 are two transactions booking the same room for same dates at the same time, If T1 starts first, in the Serializable mode, then T2 will have to wait until T1 completes. When T1 completes, T2 sees that there already exists a record which it is trying to book, so it will have to rollback. In that case, you send a message to the user, that someone already booked it.
@codeKarle I think if we need two type of queries to be supported by cassandra. let say, historical booking for a particular user and also historical booking for a particular hotel, then we need to support two cassandra cluster having same data but partition key will be different , in one it will be user_id and another hotel_id so that both can be supported. Let me know your thoughts or anyone else who also things the same
You haven't mentioned how booking service gets available rooms and room id information. Also available quantity in available rooms is heavily dependent on booking start and end date.
I have a concern here. We cannot search for hotels from ES cluster. We need real-time information about the hotel rooms inventory for searching. So hotel search should come from you booking my sql cluster. We can use cache with a small TTL for search patterns though. Thoughts?
If you think of range from 1-Jan to 3-Jan as three different requests coming for 1-Jan, 2-Jan and 3-Jan, you should be able to make that work, with the current structure easily. Now it's just about building it for each of the days in that range, and doing this in one request, which can be managed from the code+queries.
Thank you for the insightful video. Very well delivered. Question though, Where do you place users on the waiting list, who have reserved a fully booked hotel that suddenly becomes available?. Do you need a queuing system or they can be placed in the archival
thanks for detailed explanation. I am new newbie to system designs - Could you please clarify if Hotel Service, Search Services, booking service,.etc. are different EC2 machines and there would be multiple load balancer supporting each service to scale? thanks in advance.
At a very high level, kind of yes. They are different services, and by that it means they are different logical components, and it's better to assume that they would be running on different hardware. But that's not a requirement. You can have two services running on one machine, and specially with the increased adoption of container based technologies, it's more common to have multiple services running on same machine. By machine I here mean both VMs and physical hardware, and in case of EC2 you could still have two different EC2 VM instances running off the same physical hardware and multiple containers of your services running on each of those EC2 VMs. For the Load Balancers, you could have multiple loadbalancers supporting each service, or you could choose to use a Service Mesh which is a more common approach in the container based world.
I just have one doubt. In happy case -when let's say payment time expires, we get a callback and we remove the key and increment room count. What if redis goes down before the key got expired? How do we handle that case?
I think maybe we can use queues with visibility timer. That whenever a booking takes place, we add a message to queue, to expire a bookingid with a specific time that message has to be visible let's say 5 mins. Even if redis goes down, this will act as our fallback.
Great turotial. The one thing which I feel is missing : Once the hotel owner adds his hostel to the service, there should be some Map segment kind of thing that should happen in order to map a hotel to a place. I mean this can't be a general search like Amazon search, if a user searches for a place, how're we going to show all the hotels in that place? For me the segment approach which you showed during Maps tutorials would work best here.
That's a great suggestion. One possible solution without breaking this design is to add some tags while adding the hotel and in that service we could also suggest some tags to the owner so that if someone searches hotels near Parvati river we get this suggestion.
Very good video. Ref the last part in terms of dividing dcs by region, it may not work if a user in USA wants to book say a hotel in India. You may want to rethink this to be an active active infrastructure with multi leader replication and IP any casting for low latencies.
Whenever I see your videos, it feels like I'm sitting in ofc design review meeting with a senior engineer as you don't throw fancy words, really appreciate your content _/\_
Whats the drawback of having two separate databases for hotels and booking? Can you mention problems that might arise due to consistency issues and how to resolve them?
how the ordering of message maintained across many topics because many producers are publishing to the topic for a given hotel_id. How hadoop is consuming such messages considering the order of events. May be it is out of scope to discuss here about analytics.
Nice video bro, BUT i have a queation about available room table, so a room would be available for most of the future dates except a few booking, so dont you think instead of making huge number of entries for same rooms for different future dates, lets not store in this way instead we can write sql such that all the dates except the alreqdy booked dates would be automatically available??
Thanks brother .. it really helped.. especially hatsoff for explaining each of the implementation, requirement, user case etc... Please make next videos more lengthy so that beginners can understand more details.
Amazing Video thanks for sharing this ,Can you please add Summary for all your design videos , I see you have added for a few which gives a lot of sense . Thanks Again for all the great work 👍👍👍
I really like your unassuming way of explaining the essence of your topic. The only suggestion I make is that make it a little more relaxed and casual. I would like to see the tension fade away.
To answer it in a generic way, lets say Service A is calling Service B for something. Now if A needs the response of B to proceed further with what it was doing, or if they collectively form a Transactional System, then we can use Rest/RPC/any syncronous method. But wherever A does not need the response of B, and this is more of an Info that B needs to do something on, then I would rather make it an Async thing and keep any queueing solution like Kafka in between. Async RPC/Async Rest could be used, but then it has it's own set of complications like who does the retry, what if server B is down, circuit breaking, sudden bursts in traffic, etc, and Kafka is taking a lot of that complexity away.
We could do that too. There is nothing wrong in that approach as well. I would prefer the redis approach, reason: since we have an easy way to implement this and it's widely used & stable, then why bother building & maintaining a timer.
Very nice work brother..it helped us in building our system :) gratitude
Glad that it helped :)
Which system/which company you used this in?
@@codeKarle an homestay app we soon are going to launch :) we are a startup.
That's great to know 🙂
@@codeKarle brother do you contacts you share i am from non it but deeply wanna to made it.
@@rohitkhurana7424 did your start up come out?
One of the most complete and perfect system designs tutorial I have seen. The breadth and detail is perfect as an HLD.
Definitely the best system design video I have seen! very detailed explanation of each part
Thanks!! Glad that you liked it!
One of the best system design videos I have ever seen. Hats off to you and the mechanism you told for handling the case of reserving hotel bookings
Your videos are by far the best system design resources I have found on this platform. Thank you for these valuable materials!
Very well explained.
Just one point of discussion here from my end:
Can we create some inventory-service and keep availability there instead of booking-service, this will help in well defined boundaries and responsibilities.
Booking service should just keep the booking information and when booking is done, it can update the availability in one single distributed transaction.
Let me know if there are any challenges/problems in the above said approach.
Excellent video overall.
Automatic scalability: How much is the application developer controlling it (scripts, architecture, etc) and how much is the vendor component/cloud service provider (thresholds, metrics, etc) itself doing it?
I see a component re-usage pattern. User Service, Hotel/Cab Service, Booking Service, Search Service - Elastic Search, etc across your videos. Great to see the modularity being put into action.
You talk about trade offs and justify design choices which is a good but also reinforcing about first principles of components. Much more than an interview question which reflects knowledge and experience.
Only disappointing thing is it is rendered by poor voice recording. Sigh!
One of the great System design videos out on youtube. Keep up the good work..
Your videos are very helpful! Please consider uploading a low level design series as well.
Great videos on system design Sandeep. Thanks!
I have one concern about the Airbnb design. You have used Kafka for multiple things and few of them don't really require or would be actually a not-so-good design.
1. Hotel onboarding: This hotel onboarding wouldn't really require an async service as I don't see this system as write-intensive. A simple sync REST API should work.
2. Booking of a room: The booking of the room by a user & removing the room from the available list should be consistent. I think here also Kafka's use should be avoided.
In both cases, immediate reflection to the other users should be handled by a write-through cache policy. Let me know your thoughts. Thank you!
1. Why is Kafka not a good choice? There can be many workflows that we can trigger whenever a hotel is onboarded, like sending a notification to customers, advertisers etc. All of that can be decoupled into a separate service and the service can consume from Kafka
Your videos are really great for improving architectural concepts.
Thanks!
Glad that you liked them.
Do share these with your colleagues :)
i love you man. Your videos help me a lot in my prep. Keep it up!!
Thanks for great System design series. One doubt w.r.t. DB design here:
I assume search API will look like search(lat ,long, startdate, enddate, # of rooms) to return list of hotels in the given lat long area. Perhaps we can use Quad Tree type data structure here if we want to reduce latency or update booking ASAP.
(1) Available_rooms table has a field for available_quantity, Will it carry 365 entries for date column for say 1 year? If not what is more efficient way to store this info and perform fast search. Also don't you think PK should be room_id+date above for uniqueness. Please correct me if wrong.
and then (2) How are we connecting Available_Rooms with hotel.
@theghostwhowalk - I had exactly the same concern as "(2) How are we connecting Available_Rooms with hotel." - Typically there should be an association of hotel in the 'Booking' DB as well. Else how for a given date range would a list of hotels appear based on room availability.
@codekarle - Sandeep can you help out here ?
@@Legendary-Akshit Yes, booking table should be connected with hotel id rather than room id
Amazing walk through. Thank you making this video.
The content is awesome, really helpful to me. Please make videos on geographic information systems like- Google Earth, Street View, Voice recognition systems like- Siri, Alexa
How is it assured that the second SQL server in the booking service has the same details as the first one in the hotel service?
Great video, thank you!
Just a question -
1. Why don't you use Redis for the other services too?
2. Why don't you use Casandra DB for Hotel service too?
3. The arrow from my SQL cluster to archival service is strange to me. How does MySQL send messages? It wouldn't be better that after booking service is done booking then send message \ HTTP request to archived service?
4. The archived service has only one purpose, just to insert data to Cassandra. Isn't it overhead and the logic should be in the booking service?
Thank you again :)
1. Why don't you use Redis for the other services too? - Usually I prefer to do this when the read-writes ratio is very high. If there are a lot of writes then there is not too much benefit of caching. Usually these decisions would be data driven based on real access patterns.
2. Why don't you use Casandra DB for Hotel service too? - Cassandra could be used here, but I believe on this kind of dataset, you'll have lot of random queries happening, to fetch hotel by id, to fetch hotels in a region, or to fetch hotels managed by a user, etc. This would be for a lot of internal tools mainly that every company has, which we did not go over int he video, but those tools are always there. Now that query pattern is not optimal for Cassandra and we'll need a lot of data duplication to build those tools, Mysql does a better hob there.
3. The arrow from my SQL cluster to archival service is strange to me. How does MySQL send messages? It wouldn't be better that after booking service is done booking then send message \ HTTP request to archived service? - Those arrows are just to call out reads and writes. I know it's a bit cluttered there, but mysql is not sending any messages.
Archival usually should be a bulk batch job, and not done for each booking, and it makes it more optimal, since you can run this at low load times though there is nothing stopping from doing it the way you are suggesting as well.
4. The archived service has only one purpose, just to insert data to Cassandra. Isn't it overhead and the logic should be in the booking service? It could be the same service in a small scale system, but at scale this would cause problems. Think of it in a way that when archival begins, it would probably shoot up to near 100% CPU utilization for a few seconds while it is running unless you are throttling. If thats in the same service, it'll impact other features of booking service. If you have it as a separate service, you can run that out of just two machines/maybe one docker container only, while booking service can scale as per your bookings traffic.
Hope that answers your questions :)
Hello I am new to system design and learnt most of what was explained. I still need some help on archival service and cassandra. Why do we need an archival Service?
Greate video! Can anyone explain to me why we need the canssandra to store the bookings that have already happened? What is reading from it?
That is for book-keeping purpose and incase a user wants to see an older booking, which happened let's say 2 years ago. That become is not being edited, so you don't need any ACID compliance there.
@Sandeep : Nice discussion. Apart from concerns(which people have already pointed out), that if we connect booking and new hotel posting with kafka, the system will become near real time for search rather than real time which I see can be improved by using SQL DB only to power the search. Also, Kafka has some retention period like 7 days , so powering elastic search from Kafka rather than the SQL DB will result in inaccuracies in listing the hotel. please let me know your thoughts on that.
I think here the intention to use Elastic search is to support Fuzzy word search and for Word search Elastic Search DB is best as it optimised for that and gives results in best Time complexity. Also, we can do it with SQL DB but the point here is scalability and when more than a 1M concurrent users tries to search at same time SQL will be overloaded. Also, kafka retention period is nothing to do with elastic search it only retains events produced.
Can you prepare the same for movie booking system? It looks similar to this, but in movie booking, user has the freedom to choose which seat to book. The DB schemas might be different.
I am trying to understand the real need Kafka here. Can we just do the same with those two seperate MySQL DB and the corresponding services?
@codeKarle please make video for 'Design metric publishing system for cloud environments', for 'Design shopping cart' and for 'Design multiplayer game'. Thanks!
Nice Video. I just have a question on inventory.
So here we are keeping the hotel rooms inventory in ELK for powering search and also in the MySQL DB (available_room table) for booking service to interact with?
Glad I found your channel - the best System Design videos, well structured, organized and easy to follow.
Quick question - what it MySQL master becomes a bottleneck, for example there is a sporting event and a spike of bookings? Read path can be scaled by adding more slaves but how about write path?
Thanks!!
One obvious way would be the usual sharding, but that complicates the maintenance process.
I would rather split the servers in multiple groups or rings, with the intent that one group maintains the data for a few countries, and in case of planned/predictable spikes, you can throw more hardware in the groups where you expect more traffic for a couple of weeks when the traffic is expected to increase. This ring would not just be on the DB front, but also for the Application since you'll need to scale up the app servers as well most likely.
I am assuming here that we know what kind of spikes would be there, but most companies these days have that kind of a Prediction Engine build accounting the social activities/historic data/etc, so that's a fair assumption I believe.
Do share the channel with your colleagues, it helps :)
@@codeKarle Thanks Sandeep!
By splitting into multiple groups, we are still looking at having 1 master per group right? Otherwise if we add 2 or more masters we are essentially sharding, right? The question is whether can the spike within one country/city cause write traffic throughput increase to the point that a single master becomes a bottleneck and won't be able to keep up?
@@alexkorzo6129 Stumbled upon this comment. In my understanding sharding is the way to go. Probably on (country, city, hotel_id)
@@codeKarle Are you talking about consistent hashing here??
To optimize writes sharding+ replication of each shard is one way to support writes and reads. Sharding however is tricky on relational Dbs. Another approach would be active active replication across the cluster with leaderless quorum
He is mentioning about redis callbacks to use when we didn't receive payment within specified time.
But what if the redis itself is down for sometime ? ( as redis is in-memory key value store )
How are we going to make a callback which will mark the booking status as cancelled and increments the available room count.
Please make a video of Amazon Flash Sale (Great Indian festival), Distributed file system, Fantasy App like Dream 11
Every video is very helpful.
@codeKarle Shouldn't we also talk about about scenario where two hotel broker trying to book same room. For eg: AirBnB and MakeMyTrip. What should be source of truth in this case?
Great question. But more than a tech problem, it's a business headache to manage, so all these companies so a slightly different thing.
Usually if the hotel has 50 rooms, they would give off 10 to AirBnB, 15 to MMT and so on. Mostly these companies pay for these rooms upfront even if there is no customer there to get those rooms all for themselves. The benefit it gives is that each of these companies say AirBnB now has exclusive 10 rooms that it works with and takes care of booking, and they could play with demand-supply maths and change the pricing to make more money.
That being said, we could still build the system for what you are recommending. It's not a big change from what we have here, but the solution highly depends on the tech capabilities of the hotels, and who maintains the master list of inventory for each hotel.
how do deal with concurrency? two users trying to book the same room. first user from 3rd to 8th and second from 6th to 10th. how this situation will be dealt with if they are trying to book at the same time?
How MySQL is horizontally scaled? as you have mentioned every component can be horizontally scalable.
Mind my ignorance, but what does LB stand for?
Amazing video, thanks a lot
Thank you for a great explanation
U a System Design God
Room availability for any future date range with elastic search was not clear.
Can you make a video on how you'd design metrics publisher system? (Based on logs, rules, etc.)
+1
@10:10 "thereby making sure that this is having a scale that MySQL can easily handle". Can you explain or write the blog post on hotel booking system design.
For DC replication , what happens if the person connects to the R2 DC (from US) and tries searching for hotels in region which is NOT maintained on the data-store for that region ?
The whole data layer needs to be replicated, I think
Have the same question
All the data will be replicated to each data center that was opening statement. Partition data just acts like multi-master. But data is still replicated. Other DC can act as slave, where reads can happen.
Wonderful!
ty
The microphone quality is extremely bad. I'm barely able to understand.
Get a new microphone, bro
Great video. I have one question though. How are you generating/updating the available_rooms table? Where's that data coming from to begin with? Thanks!
That's probably from UI/App for hotel to hotel svc right?
Really great video : Have some doubts below
- So how are we storing available table. Are we going to store for 365 for each hotel . That will be inefficient.
- Which service will pull data from hadoop cluster? once the analysis done from hadoop cluster aren't we storing it in Analytic DB?
- Should not we store images first to s3/gcs then later cdn?
Excellent points. I had the same question. I think we shouldn't write images directly to CDN. We will store them in an object storage and CDN will cache it based on usage of the files geographically.
In case anyone reads it now,
I believe we are anyways dividing the entire geography into parts in terms of data centers so it won't be exactly a single table populating the data for all the hotels and its room types for 365 days.
It will be more like -
If I divide the entire geography into 4 parts where each part will have a separate database, so available_rooms table can contain 1 million/4 = 250000 hotels * 10 room categories * 365 days = ~750 million rows. Ofcourse this data will be sharded based on hotel IDs so it will be separated into separate shards.
what if customer booking for hotel and at the same time hotel delist their room. Since the databases are different it would definitely create inconsistency. How are you planning to handle that ? Will you be confirming the room to both the databases ?
User generally spend 5-10 minutes to make the payment, and in between if the hotel is delisted then how it is handled ?
the booking table will have an entry with reserved status for that room_id, the hotel staff wont be allowed to delist
A wonderful playlist to binge watch :D More power to you Sandeep :)
Glad that you liked it :)
@CodeKarlw I like this design, BUT:
- Using a RDBMS for performing the booking lock may not be appreciated by all interviewers: I proposed a very similar architecture in a sr backend engineer interview @ Uber and I've been rejected with the motivation "He used a RDBMS for performing the reservation lock". It would be interesting seeing an alternate implementation. (Redis lock?)
- The service crash scenario is completely absent in the design, specifically a crash of the booking service after a payment has been successfully performed but before the booking status update
- How do you maintain the datacenter database aligned? Shared kafka topic consumed by an align service?
Thank you for your work anyway!
I don't understand a few points here -
1. How is the available quantity only a function of room_id. If we are accepting date range in POST /book, it should be accounted for in the schema.
2. Assuming, we have that in the schema, we should also need to have an efficient query that checks for availaibility based on these three criterias.
You can probably utilize range based locks while reading the available state and creating the booking entry.
Superb Sandeep. I am looking for Coupon management system design. its commonly asked across companies. On internet, there is not a single design for this.
Can you please create a video focusing only on the "search hotels". I am sure that will be interesting
I'm wondering how we can get quick Search results with filters like available_quality, date_range and room_classification.
What happens if the booking is for 10 days? It will result in 10 queries to available rooms table to find if the room is available for that day ?
No. We'll do one query where we fetch the data for all the dates. Batch query is usually a better idea than multiple queries
Can't even understand what he's saying, very poor audio..
Great video Sandeep. Quick tip for Product Managers looking at this - I think the first 15 minutes and then 32:20 onwards is pretty relevant.
Wanted your thoughts on something:
The safest way to interact between the booking service and the search service would be to have a single DB where locks are taken on the respective fields when a booking is made. This would guarantee that no room is double booked ever. That approach seems very very 90s and theoretical. But is there ever a case in your design that the delay between syncing the search and booking service through Kafka can cause a double booking scenario. Are there any other cons you could think about this architecture?
Thanks in advance.
Hello, we are using only one DB for booking management. There will never be a case where there would be a double booking. there can be a case where in search experience hotel is shown as available but on trying to book - it might be shows as unavailable. But this should be in only very minimal cases
Nice Explanation , I am having one doubt here , From Hotel Service Storing the data into one MYSQL Cluster, and from Booking Service Storing the Data into Another MYSQL Cluster, then How do they in sync ? I am mean Hotel Service data and booking service data in DBS.
Thanks for article.
Live updates to the Hotel will get published to Kafka and it feeds into Elastic Search. But my question is, Elastic Search only holds live updates or its like a replica of Hotel DB ?
For the issue where TTL expires before the Payment success message, why can we set the TTL long enough (more than Payment service SLA) for us to be absolutely sure the payment will never be successful later.
Con is definitely that the room is reserved for more than it should be blocked for, but we definitely reduce some complexity there.
Kudos for the effort
Good video. Just one concern, how come room id will have quantity, I think room id represents one room and quantity should always be one. We should have another table which maps room id to booking id.
Thanks for the great explanation.
Question: Why do you suggest the approach of setting the boolean "isActive" to false, to delete a record rather than using the SQL delete operation?
It's generally not a good idea to delete something from the Database. If you need that information for auditing, financial reporting, or just debugging, it'll be helpful to have that information in our DB marked inactive. Additionally delete queries so have a risk of deleting something accidentally in case of bugs, so i'll always prefer to have a delete/isActive flag and filter the active records in get queries.
Hi Karle, I have one question. Will Kafka only contains all the hotel/room data, same as a database? If we treat Kafka as a source of truth for the input of search service, and some room data that exist in the database are not in Kafka, the search result will always miss that part of room data. In addition, the Kafka data will be expired and evited in 7 days, which means some room data will not be able to search by end-user to book, even though they are available, just because it does not exist in Kafka, and we treat Kafka as the source of search service. Could you help to clarify it? Thanks.
It isn't Kafka where we keep this data. We will keep this in ElasticSearch. Kafka is just a message broker and passes all the info from sql db to elastic search for better search capabilities.
Great video. One thing i that i found missing in this design is :
When admin of hotel adds new rooms -> there is no flows that updates rooms availability in DB of booking service
Please let me know if i i'm missing anything on this
Can anyone help me know why SQL Database(MySQL) is used in Hotel Service Application, wouldn't having NoSQL be correct choice.
I see a point on using SQL in Booking Service.
For hotels data store images in s3 or any blob storage and metadata in MySQL with ref to s3 as a link and then have cdn pull data from s3 which can be accessed via the user app along with metadata for the hotel. Not sure why cdn is being used as store for images in this design..
For hotels data store images in s3 or any blob storage and metadata in MySQL with ref to s3 as a link and then have cdn pull data from s3 which can be accessed via the user app along with metadata for the hotel. Not sure why cdn is being used as store for images in this design..
the way you have linked CDN to hotel service is wrong. The way CDNs work is they have edge nodes (the one that you have represented) and origin servers. The CDN is always seeded by the origin server when the client requests the data. You don't upload to CDN or make any modifications to the CDN as shown in the diagram.
Contentful video,very nicely explained .. Basically i was looking out for scenario , where multiple people select the same hotel almost at the same time , or like how we do solve the race condition in such a huge system ..? by avoiding same booking with multiple people.
The property ISOLATION of ACID properties deals with that. So, if you set the Isolation level of a transaction (T1) to be Serializable, no other transaction would be able to write into the tables involved in the first transaction (T1). So, whichever transaction starts first, that transaction will end first. In the case of a booking scenario, let's say, T1 and T2 are two transactions booking the same room for same dates at the same time, If T1 starts first, in the Serializable mode, then T2 will have to wait until T1 completes. When T1 completes, T2 sees that there already exists a record which it is trying to book, so it will have to rollback. In that case, you send a message to the user, that someone already booked it.
@codeKarle
I think if we need two type of queries to be supported by cassandra. let say, historical booking for a particular user and also historical booking for a particular hotel, then we need to support two cassandra cluster having same data but partition key will be different , in one it will be user_id and another hotel_id so that both can be supported.
Let me know your thoughts or anyone else who also things the same
You haven't mentioned how booking service gets available rooms and room id information. Also available quantity in available rooms is heavily dependent on booking start and end date.
Availability information should come together with search results - you need to know price and number of available rooms beforehand.
I have a concern here. We cannot search for hotels from ES cluster. We need real-time information about the hotel rooms inventory for searching. So hotel search should come from you booking my sql cluster. We can use cache with a small TTL for search patterns though. Thoughts?
@18:45, why would there be available_qty for a given room for a given date. Doesn't make any sense at all.
@codeKarle
Why do we need interaction b/w archival service and kafka queue as per diagram?
Any reason for that
Also do not see an api gateway in front of the lb which could then load balance the distributed services..
Thanks for the video, I do not quite understand is HotelService database will have the same data as in ES Cluster?
The availability room table just has a date table. I did not understand, how were you able to book for date range.
If you think of range from 1-Jan to 3-Jan as three different requests coming for 1-Jan, 2-Jan and 3-Jan, you should be able to make that work, with the current structure easily.
Now it's just about building it for each of the days in that range, and doing this in one request, which can be managed from the code+queries.
Thank you for the insightful video. Very well delivered. Question though, Where do you place users on the waiting list, who have reserved a fully booked hotel that suddenly becomes available?. Do you need a queuing system or they can be placed in the archival
How does checking for room availability work from your design? We could have multiple bookings for the same room for different dates.
one question here.. if room is not available it won't come in search right. Then why we are checking in the booking service ?
have cache on top of mysql and use lazy loading for EC, instead of kafka for search service
thanks for detailed explanation. I am new newbie to system designs - Could you please clarify if Hotel Service, Search Services, booking service,.etc. are different EC2 machines and there would be multiple load balancer supporting each service to scale? thanks in advance.
At a very high level, kind of yes.
They are different services, and by that it means they are different logical components, and it's better to assume that they would be running on different hardware.
But that's not a requirement. You can have two services running on one machine, and specially with the increased adoption of container based technologies, it's more common to have multiple services running on same machine.
By machine I here mean both VMs and physical hardware, and in case of EC2 you could still have two different EC2 VM instances running off the same physical hardware and multiple containers of your services running on each of those EC2 VMs.
For the Load Balancers, you could have multiple loadbalancers supporting each service, or you could choose to use a Service Mesh which is a more common approach in the container based world.
I just have one doubt. In happy case -when let's say payment time expires, we get a callback and we remove the key and increment room count.
What if redis goes down before the key got expired? How do we handle that case?
I think maybe we can use queues with visibility timer. That whenever a booking takes place, we add a message to queue, to expire a bookingid with a specific time that message has to be visible let's say 5 mins. Even if redis goes down, this will act as our fallback.
This channel should have millions of subscribers.. you are highly under rated. Keep doing the good work
Thanks!!
Can't get there without your support.
Do share the channel with your friends/colleagues and hopefully we'll get there :)
how the supply demand works here is not a scope of this video, but would like to know.
You have a knowledge, topic is interesting but I can barely hear you!
Great turotial. The one thing which I feel is missing : Once the hotel owner adds his hostel to the service, there should be some Map segment kind of thing that should happen in order to map a hotel to a place. I mean this can't be a general search like Amazon search, if a user searches for a place, how're we going to show all the hotels in that place? For me the segment approach which you showed during Maps tutorials would work best here.
That's a great suggestion. One possible solution without breaking this design is to add some tags while adding the hotel and in that service we could also suggest some tags to the owner so that if someone searches hotels near Parvati river we get this suggestion.
why do you need Mysql for Hotel? Why couldn't it be Nosql like Mongo?
Very good video. Ref the last part in terms of dividing dcs by region, it may not work if a user in USA wants to book say a hotel in India. You may want to rethink this to be an active active infrastructure with multi leader replication and IP any casting for low latencies.
Whenever I see your videos, it feels like I'm sitting in ofc design review meeting with a senior engineer as you don't throw fancy words, really appreciate your content _/\_
10 000 000 / 500 000 = 20 average rooms.. 1000 seemed too much, I had to calculate this :D
Whats the drawback of having two separate databases for hotels and booking? Can you mention problems that might arise due to consistency issues and how to resolve them?
how the ordering of message maintained across many topics because many producers are publishing to the topic for a given hotel_id. How hadoop is consuming such messages considering the order of events. May be it is out of scope to discuss here about analytics.
maybe maintaining message_key for every event can help the cause. events with same message_key (here hotel_id) will be pushed to same partition.
since every hotel has different types of rooms, and each room has various different attributes, why not use mongo db?
Database choice should never be driven by such small concerns. Every database these days handles unstructured data.
why do we use kafka vs just writing to the db from the hotel service?
Its amazing how you have utilized Kafka to interact with different systems. Can you clarify- what if kafka is failed? how are we handling that
Then all hell breaks loose. It’s assumed Kafka has high availability with partitioning replication
Nice video bro, BUT i have a queation about available room table, so a room would be available for most of the future dates except a few booking, so dont you think instead of making huge number of entries for same rooms for different future dates, lets not store in this way instead we can write sql such that all the dates except the alreqdy booked dates would be automatically available??
Thanks brother .. it really helped.. especially hatsoff for explaining each of the implementation, requirement, user case etc...
Please make next videos more lengthy so that beginners can understand more details.
Amazing Video thanks for sharing this ,Can you please add Summary for all your design videos , I see you have added for a few which gives a lot of sense . Thanks Again for all the great work 👍👍👍
Thanks!!
Yeah, that is work in progress. Other summaries would be live soon in a couple of days at www.codekarle.com/
I really like your unassuming way of explaining the essence of your topic. The only suggestion I make is that make it a little more relaxed and casual. I would like to see the tension fade away.
Very nice design presentation. Question: why do you use kafka instead of just RPCs, say, between the hodel svc and booking svc?
To answer it in a generic way, lets say Service A is calling Service B for something.
Now if A needs the response of B to proceed further with what it was doing, or if they collectively form a Transactional System, then we can use Rest/RPC/any syncronous method. But wherever A does not need the response of B, and this is more of an Info that B needs to do something on, then I would rather make it an Async thing and keep any queueing solution like Kafka in between.
Async RPC/Async Rest could be used, but then it has it's own set of complications like who does the retry, what if server B is down, circuit breaking, sudden bursts in traffic, etc, and Kafka is taking a lot of that complexity away.
very nicely explained. Thank you
Question: why can't we rely on our timer in payment service rather than getting notified from redis?
We could do that too. There is nothing wrong in that approach as well.
I would prefer the redis approach, reason: since we have an easy way to implement this and it's widely used & stable, then why bother building & maintaining a timer.