Netflix System Design | System Design YouTube | System Design Video Streaming App
Вставка
- Опубліковано 2 гру 2020
- System Design | #SystemDesign :
Netflix system design is commonly asked interview question based on OTT system design.
In this video I have explained how you can come up with system design for video streaming applications like youtube, netflix, hotstar or amazon prime.
It is commonly asked in Design Interview Round of Big Tech Companies including FAANG.
Recently it has become a very common systems design interview question due to the fact that, uploading a video and providing a seamless streaming has a trick involved which netflix solved in a brilliant way.
Video covers following requirements:
1) How Netflix, youtube etc onboards a video
2) What is a cdn and how it helps in reducing buffer time
3) How to search for a videos
4) How user experience is taken care of by streaming chunks based on the bandwidth and device of user.
You can buy us a coffee at: www.buymeacoffee.com/thetechg...
system design (HLD and LLD): • System Design | Distri...
DS for beginners: • Arrays Data Structures...
leetcode solutions: • Leetcode 84 | Largest ...
github: github.com/TheTechGranth/theg...
facebook group : / 741317603336313
twitter: / granthtech - Наука та технологія
All videos in this playlist are a gem! They have the perfect level of detail to understand system design for someone with technical functional knowledge and skillset. Seen so many similar channels so far, but this is the best. Great work!
Amazing explanation. I rarely comment on youtube, your explanation pulled me to write down a comment and appreciate your hard work.
Excellent video. Great explanation. I'm subscriber of your channel.
Thank you for your contribution to the community
The short and crispy explanation ..👌
Very detailed design. Thank you for this
Its a great video for understanding complete basic streaming concepts.
Glad it was helpful, do like and subscribe and share with others 🙂
Great video.
Have small doubt are we going to use same queue does it not cause single point of failure if the queue goes down.
Just wondering how can you cache on ISP? I guess it should be DNS.
Very Neat.!!.Great content !! Please keep uploading more on arch design.
Glad it was helpful. Do like and subscribe and share with others 🙂
Great video, wondering what the event that gets sent into the Kafka queue would look like.. Is it just an event with the bucket and file name and then each of the microservices that reads off the queue makes a call to the bucket or is it data like a byte array that is gone into the queue and it uses that to split and encode the data ?
Your 1st option is what I had in mind and makes a more decoupled system as per my understanding. So the event can be like a message payload in form of XML or json, with the details of task to be performed, e.g video name, chunk name etc. And then based on these data each microservices can perform their job
You made my day brother 🙏🙏🙏🙏🙏
Did you mean DNS when you said ISP in the view video section?
Few points:
The content verification service should be done before encoding because if we want to remove the video for violence or something, then we would have wasted the compute for encoding unnecessarily
Great content! Thanks
How would the logic work for how the Video Distributor Service decides to which cache locations to distribute the content? Would that service make requests to some pre-trained ML model that uses all the metadata as input? That model can be trained based on how certain metadata corresponds to the popularity of certain movies, and restricted based on country-specific licensing or content restrictions. Also, any articles about this particular service?
I am not sure if Netflix or youtube uses model training but if you have enough data you can come up with your own algorithm for recommendations. Based on the recommendation collected for multiple users who are using a CDN, video distribution can be done.
You can check out this article on Netflix page:
about.netflix.com/en/news/how-netflix-works-with-isps-around-the-globe-to-deliver-a-great-viewing-experience
@@TheTechGranth Thanks! Yes, I see what you mean. The article says that the distribution and recommendation algorithms have similarities:
"Although the number and size of the files that make up our content library can be staggering, we are able to use sophisticated popularity models to make sure the right file is on the right server at the right time. These advanced algorithms share some common approaches, and sometimes common inputs, with our industry-leading content recommendation systems."
@@YannStoneman And it makes total sense, if you recommend something and there is a lag in displaying that video, think what kind of user experience it will have?
How are the video chunks stored? How does the encoder know where to look for each chunk for a single video?
1) same storage where original video was stored
2) there will be a metadata table which will map chunks to original video along with resolution, format and url for the each chunk
Great content !! Please keep uploading
Thank you, hope it was helpful!!! Do share it with your friends and ask them to subscribe :)
Is a web socket connection established for uploads? Since file size can be very large?
It can be but even http will work
Can you please clear why we are storing the meta data 2 times in database one after storing in Distributed storage and once after encoding the chunks.
Why we are not putting in meta data once after encoding part is done of the chunks
It will not be duplicate actually, 1st time you will store all the details including tags, description, cast etc. and url of the original video in distributed system. 2nd time you will store the details of the chunks, like mapping of chunks to the main video that was uploaded, ordering of chunks etc.
I mean this is completely how I would design a streaming app like this. This is in, no way what actually youtube or netflix does. They have a lot of complex algo under the hood.
These videos that I create, I do the high level designing keeping in mind that we have to explain it to interviewer with an hour time.
Very well explained
Glad it was helpful. Do like and subscribe and share with others 🙂
hi sir, what do you mean by distributed storage ? you are saying that its present in Europe but if its distributed then its present everywhere. why we require cdn 's then
Great content. Can you please make video on explaining Distributed logging
Yes, I need to work on that video, there was one more request for that. I have couple of more videos in line for system design, and post that I will pick up distributed logging
Here you go: ua-cam.com/video/WzHgOl3xvu4/v-deo.html
Wonderful vedio, could you also share how the vedio is stored in cache, and how retrieval is made fast from there. Are we storing all formats of chunks against some key. what key will be used here?
Wonderful ....
Voice is clear....
Thanks, hope the content was helpful. Do check out the other videos. Kindly like and subscribe and share with your friends, it really motivates :)
@@TheTechGranth Wonderful content ... It gives a high level overview of a streaming service like youtube/netflix .... But the challenge here is microservice scale and deployment ..... Many of the current commercial roadside dev. guys fail at scale .... scale is the business mantra of any ecommerce or streaming platform.
I have some plans of starting a video based collabration platform between students and teachers. This platform is going to be state of the art where a teacher/student can create doodly type animated videos and upload on this platform. It allows students to create groups and do self study and brianstorming. It would of great help to the the poor rural children to learn & understand concepts better in thier own language ......
@@zameerahmed1775 That is really great initiative. Here is the content on microservices you can implement some patterns mentioned along with service discovery to scale your api. When it comes to db you can take hybrid approach like the ones used in instagram video
@@zameerahmed1775 ua-cam.com/video/XpIMnFpVrUg/v-deo.html
Nice video. Could you also tell how the TV shows are made available in hotstar applications. When a new movie is uploaded all the process which you explained I could relate to but could you also tell how the day to day tv shows which we see in our TV are made available in OTT applications?
For tv shows, it would still follow the same procedure, but do you by any chance mean live streaming?
@@TheTechGranth I meant the normal TV shows and you have already replied that it follows the same procedure.Thanks :) It would be great if you can give details on what protocols are being used in these OTT applications.
@@lakshmic3630 Please keep in mind that this is the design that I came up with and not the actual design what netflix or any OTT uses. for communication, all the internal ones which are based on events from kafka will work on publisher-subscriber model. User will be connected via websocket because that communication needs to be 2 way communication, as based on bandwidth of user there is a chance that you need to send a lower or a higher quality video.
Internal microservice to microservice can be rest calls.
I hope it was all helpful. Do like the video and subscribe. please share with your friends and ask them to like share and subscribe, which will help me grow :)
Thanks,
Ayush
@@TheTechGranth Thanks for the detail :) Sure will do
Great Video!!
One Doubt:
Say, in the Original S3, the links for
Format A Resolution 480 Chunk X is L1
Format B Resolution 1080 Chink Y is L2
And so on....
Doubt 1:
What will be the link be after storing them to the Cache (open connect) at ISP?
Doubt 2:
How will viewer client get to know know URL?
Doubt 3:
The Database has the links to L1, L2, L3...of the original S3 storage. How will it fetch the links saved in the OC(cache at the IPS)?
What is the flow? Like: Client - ISP - fetch video service - DB - ISP - Cache - client ?
Thanks for video.
Can you make system design video for payment gateway please ?There is no video on this problem statement
any reference to any video or article you know is also appreciated?
thanks in advance
Great suggestion, Thanks , let me add it to my list. I will try to come up with this!!!
@@TheTechGranth Sure , I am eagerly looking forward to it.
Thanks a lot in advance
you can check out this: www.lyra.com/in/what-is-payment-gateway
@@kunal4350 here you go for payment gateway system design
ua-cam.com/video/NxjGFIgFCbg/v-deo.html
Do like and share 🙂 and subscribe, if not already 🙂
I don't believe all http packets takes support of TCP. Mostly UDP suppors streaming for video streaming
Everything other things are correct. Additional to your elaboration, for providing guaranteed video streaming SCTP where stream large data is divided to data chucnk with index at transport- segment which helps with reassembly at receiving device much coherently and guaranteed assembly interleaved of data chunks happened at segments encapsulated in packets.
Cant we move this video splitting task to client side as this will have 2 advantages so to speak:
1. This will make the upload faster due to smaller chunks.
2. In case of failure we just need to upload the failed chunk based on content addressable storage, saving the bandwidth as well.
Please let me know of your thoughts.
What if the processing capacity of client is bad?
What happens when storage of client is low?
Both design will have it's pros and cons
@@TheTechGranth Processing capacity won't have much impact IMO as uploading video is not a time critical scenario as of streaming videos.
Storage should also not be a concern as file is divided into chunks and then uploaded one after another.
I agree both have pros and cons but we need to choose having less cons :)
@@uditagrawal6603 he is saying about processing the video and converting to smaller chunks.
how can converting to smaller chunks reduce time. at the end it has to upload same size of data. maybe parallelizing the upload from different mediums can.
next thing about storage size is that video processing scripts do require memory space which may not be sufficient.
In Africa lot of people watch bollywood movies. So we can add bollywood movies to Africa CDN as well
Bollywood suppose to be the name of the Hollywood in Africa
Nice neat video! Code Karle - ua-cam.com/video/lYoSd2WCJTo/v-deo.html offers a in-depth video on UA-cam system design, whereas you offer is slightly simpler version. Which one is expected in the interview for mid-career positions (3-5 yoe ?)
I have not watched that video but if you have watched both then you can pick up the gist and good points from both and that should be your answer 🙂 my intentions were to cover an overall picture of Netflix. In an interview you might be asked to pick one of these component and interviewer can drill down in that. So if you know the core concepts that I have explained you should be able to answer those questions, do let me know if you have some doubts and ofcourse like and subscribe and share with others 🙂