Instagram System Design | Facebook Feed | Promise Based Cache | Feed Generation Design
Вставка
- Опубліковано 8 тра 2021
- #SystemDesign
System design interview preparation:
In this video we have designed Instagram. Instagram is a photo sharing app. The design mentioned here for user feed generation can be used in a system like facebook and twitter time line generation as well.
Topics discussed here are:
1) requirement gather
2) capacity estimation
3) database design for the table
4) how to shard your tables
5) which database to use and design considerations
6) pre computation of user feeds
7) likes aggregation, specially in case of celebrities where aggregation happens too soon
8) promise based cache to save db from getting bombarded with requests in case of cache miss
Overall I have tried to cover everything from table design to API to components including the design consideration for each case.
URL shortner used as part of design : • Video
You can buy us a coffee at : www.buymeacoffee.com/thetechg...
system design: • System Design | Distri...
DS for beginners: • Arrays Data Structures...
leetcode solutions: • Leetcode 84 | Largest ...
github: github.com/TheTechGranth/theg...
facebook group : / 741317603336313
twitter: / granthtech - Наука та технологія
I think it is better to store comments/likes for a post inside the post itself. comments/likes cannot exist without the post. when the post is deleted, all other things need to be deleted. Moreover, comments/likes are not searchable on any platform (for a reason of course).
In the cache we may only store the number of likes and top few comments. When the user hits "show more comments" then we actually hit the post db, to find out more comments on that post. To delete a comment/like, user can pass us the [postID, commentID] from the UI. what do you suggest ?
post = {
id:...
...
comments:[ ],
likes: [ ]
}
Hands down prob one of the best system design videos out there. Covered everything so elegantly. Highly underrated. I'm subscribed!
Hope it was helpful. Do share with others 😀
Finally got nice content for Instagram System Design. Covered most of the point. Really helpful. Thanks
Glad it was helpful. Do like and subscribe and share with others 🙂
Very well explained. Your effort is greatly appreciated
I am not sure why this playlist is not popular. The most crisp and to the point content. Thanks a lot for your efforts.
Also it would be great, if at the end of the video we can get some real world solutions orgs are using. Like you mentioned about promise cache which Instagram is using, Similarly I think there is a gossip protocol SWIM which Uber uses.
Also to add, for comment and likes count we can take help of a fantastic DS introduced by Redis, HyperLogLog. You can have a look at it, and maybe show us one use case in your next video.
Liked the way you explained the feed generation and 'like' aggregation logic. Also, it was overall very detailed design with lots of information. Thank you!
Glad it was helpful. Do like and subscribe and share with others
Very pratical and details , Thanks man
Man this is one of the best System design videos I’ve seen in a while. You deserve to become more famous 👌🏼
Do share like and subscribe 😀
Got an overview on the System Design, Capacity Planning and soon... Thank you.
Glad it was helpful, do like and subscribe and share with other 🙂
Good to know about promised based cache! thanks
Glad it was helpful. Do like share and subscribe :)
Learned few new things here .. FAN OUT SERVICE , PROMISED BASED CACHE.. Thanks for such detailed explanation..! Keep it up.
Glad it was helpful. Do like share and subscribe 🙂
@@TheTechGranth promise based cache was icing on the cake. Thanks. Instagram recently mentioned in one of the tech talks.
Hey man great video and thanks for explaining Instagram design in simple terms. Just one small suggestion: In the ending section of the video, it would be super helpful if you can also show the final architecture diagram to grasp the full picture once again, just like we do on the whiteboard when we are done providing the solution :)
I have seen many instagram design videos, this one is better than others
I see the same comment under every other video.😂
Also instead of having all data in tables, i guess we can have a combination of RDBMS and NOSQL dbs. That will be much faster. But it’s my opinion I may be wrong. On the positive side, i really like your videos they are really insightful.
Yes it can be split in rdbms and nosql based on exact requirements we are going to meet.
Thanks a lot, hope these are helpful. Do like and subscribe and share with your friends
Great video!
I wonder why this video has so low views or likes...It is very well explained apart from the estimations :)....way better than those overhyped channels...way to go bro...this helped me a lot...thanks for your hard work!!
Glad it was helpful to you :) Do like and subscribe and share with others. It might help the views and likes 🙂
Thanks for this wonderful session! I have one question regarding sharding, Can you explain how is sharding by postid efficient ? data would be loaded equally but incase when we want to query all post of a specific user we may need to query multiple shards rt?
One of the most detailed system design videos for Instagram. I wasn't much aware of the fan out concept before this video. Thank you.
I have a question though. At 32:35, you said that we will only be reading a small number of columns for which a columnar db makes more sense. But lets say the Post table consist of post_id, user_id, caption, created_on, image_link. So, wouldn't all this information be required? I mean we should show the author, image, caption, created_on etc along with the post in the User feed(The same happens in actual Instagram too). So, why are we saying that we only need to read "some" columns in majority of the cases.
I understand that it might be difficult to scale RDBMS at such large scale through sharding but other than that, the only reason I can think of for not using RDBMS is that we need partition tolerance and availability for which cassandara might be a better choice. Am I missing something else which might indicate as to why we shouldn't use RDBMS?
5-6 columns are small set of columns here the major problem arises when we have to aggregate stuffs like number of comments and likes for each post for each user. This is where columanar store will do its magic 🙂
Just woww
Do like share and subscribe :) and check out other videos on system design hld and lld
two questions:
1) why not using some other cache service separately for celebs?
2) why not storing the count of likes in post tables and separate like on other tables so when user will like it will increament the like in post table also insert like in like table!
Very nice video boss.
Gald it was helpful. Do like and subscribe and share with your friends 🙂
Why user and follower tables are mySql and other tables that are related to post service in Cassandra?
What is the user of like_id in like table? If we want to generate who liked a post, then shoudln't we have posters's userId nd likedUserID? That way se can query which users liked the post. Kindly correct me if am wrong.
Please make a separate video the functionality of columnar database. In what use cases, it is advisable to go for the same?
I have already come up with a video on, how to choose a database for you system. Have covered the requirements you are asking:
So like share and subscribe 🙂
ua-cam.com/video/leGv3PIaCn4/v-deo.html
How the Url Shortner service will save the space in this case for photos ??
How do we ensure practically that few instances of a service are for writing and few instances are for reading
That is a great explanation. But, How will you handle the user viewed feed/post. If user is scrolling the feed fast.We need to track that user view this post.So, we don;t show the same post/feed again.
In the post table, you mentioned sir that photoURL will be the path to photo in S3, but a single post will have mulitple pics/videos each of which will have unique photoURL from S3 na?
Is one table enough for the "follow" part. Some videos suggest two tables : one for the followers(user's followers) and another one for followings(the people who the user follows). It seems one can serve the purpose of both. Not sure if there can be any advantage of having two seperate tables. Any thoughts?
Query to pick up the following part seemed simple and straight forward to me, plus the way we shared the data, would be able to handle the query load. Duplicating data makes sense in case where we have significant performance improvement, here I do not see any such thing
From where did you learn in such details?
stilll not sure why we need a rdbms as you mentioned in the earlier part of the video. can you elaborate on that in detail please? have a loop coming up haha
It is just to store the relational data. When it comes to photos and videos you have to store the metadata for these as well
Check out this:
ua-cam.com/video/leGv3PIaCn4/v-deo.html
can you explain why we need elastic search and mysql db. Can't elastic search handle all the operations.?
Elastic search has it's own capabilities and cons. When dealing with structured and relational data, I would always prefer a rdbms over a no sql database
This is my take on choosing database
ua-cam.com/video/leGv3PIaCn4/v-deo.html
@@TheTechGranth Hi thanks for your reply. I am confused in the explanation why we need both elastic search and mysql for storing same data. Are we n't doing duplication. if we need to use elastic search for string based search query cant we use it for searching a particular object by it's id. I am not much familiar with elastic search so please feel free to redirect to some link if that can be helpful.
@@jainso I thought of that to optimize the search api and the user api both, yes data duplication is there but trade off is faster response time and consistent user data.
This is my thought process
@@TheTechGranth thanks for your quick response.
@@jainso You are welcome. Do like and subscribe and share with your friends 🙂
The storage you calculated in the beginning is on the image n video. Also you are saying that image storage will be in s3 that means 973 gb will not be in db. Db will have only metadata. The data in database will be high because of lot of users and we need to shard I agree with you but it will be less than 973gb as image/video storage is separate.
The estimation I added here was just for image storage, metadata and users will be in db and yes size will be more
@@TheTechGranth 970 Gb estimated was for images. They are being stored in S3.
We need to have additional estimations for the Db. Right ?
Also, can we use graph db for representing connections or follower relationships. ? Will there be CDNs present for storing the images/reels.
Can you make notes on reconcile service for computing the likes. Need some clarity on the flow
@K V V yes you are correct regarding the estimate.
Graph db may not be required here as the schema will be straight forward, you can check the Instagram reels system design video for the likes and db part.
@@kvv6452 ua-cam.com/video/OPo_FB35E04/v-deo.html
one doubt what is the use of like_id in like table?
It is just the primary key for that table
1. The distributed cache is storing lots of data for 100M users. is there any limit on the cache. I know it is storing only meta data.
2. The bottle neck is distributed cache if it is failed application will be slow down.
3. can I store the 6 month's data like post, comment etc into MySql ( RDBMS) post that means after 6 months old data to NoSQL DB.
Kindly help me to understand above points
Hi Tech Granth,
The hybrid approach for sending News Feed contents to the users: We can move all the users who have a high number of follows to a pull-based model and only push data to those users who have a few hundred (or thousand) follows.
Plz, update the video accordingly.
That is what I explained at 34:30
@@TheTechGranth ok thanks
@@rishirajtandon3849 hope it was helpful. Do like and subscribe and share with your friends :)
If the user sees the post, how to maintain that we don't show user that post again? How you will be storing that post in cache. Can you give some concrete design.
Prepend post in timeline, based on post time
@@TheTechGranth not always as we sometimes give more priority to post due to user preferences and behaviour, prioritise post based on what user like, rather than just time stamp.
@@FWTteam Got your point, this won't be a simple post in that case, you need to run some analytics 1st to understand the likes and behaviour, which can then be fed to some ML model for assigning real time priority. For example Insta reels, where you are shown reels according to your liking also this can be done for recommended posts and not the post from a friend. For post shown on timeline, which belongs to friend, it will always be in chronological order
I have a few questions
1. when we duplicate the storage to make the system fault tolerance, shouldn't there be multiple copies of db instances?
2. Usually in this problem, we will search the system by username first and then we will dig into their posts, so if we shard on post id, will the queries be faster?
🎉
0:23
30:29-36:00
1:41 Parabéns AUTORIDADE SOCIAL por me ajudar a conquistar um público mais amplo e aumentar sua presença online por meio de estratégias eficientes de impulsionamento em redes sociais.