How Pinterest Scaled to 11 Million Users With Only 6 Engineers
Вставка
- Опубліковано 23 лип 2024
- In this video, we will be exploring how Pinterest managed to scale to 11 million users with only 6 engineers. We are going to explore their early tech stacks, issues they ran into with scaling and how they overcame those obstacles.
Original article link: highscalability.com/blog/2013/...
Timeline:
00:00:00 - Intro
00:00:10 - Background story
00:01:48 - What is clustering?
00:03:20 - What is Sharding?
00:03:37 - Sharding prereq work
00:04:04 - Issue 1
00:04:42 - Transition to sharding
00:06:35 - Issue 2
00:07:14 - Other issues
00:08:01 - Takeaways and ending
Credits:
- Copyright Music] Chill Lo-fi Hip Hop Beat FREE Instrumental (Copyright Free) Chillhop Music: • [No Copyright Music] C...
- (no copyright music) lofi type beat “biscuit” | free vlog music | prod. by lukrembo : • (no copyright music) l... - Pieces - After The Fall - • Pieces - After The Fall - Наука та технологія
This video proves two important things to me:
1. Quality is far over quantity in hiring for engineers.
2. Simplicity principles like KISS is a blessing in any form of engineering
@yestechguy-fj9tm Depends on work in quantity vs quality.
@yestechguy-fj9tm im a software engineer and sometimes there are alot of features to implement which are pretty basic. Ofcourse designing a scalable application isn’t one of them
agreed and even more critical if you're just starting out!
The less 'moving parts' there are, the less prone to failure something will be.
Absolutely, the book The Mythical Man-Month written by Fred Brooks talks about this, I recommend a lot reading this book, it helps you prevent erros that when you know it, it seems obvious, but when you don't know, it leads to a catastrophe
As a very new developer... I do not understand many things he is saying.... I am just amazed by the ingenuity of engineers... I hope to be like one one day
read the first few chapters in Designing data intensive application
Heyy don't worry if you don't understand these concepts as you're starting out! I honestly didn't know much myself when I first started. Had to spend a lot of time studying system design for interviews and that's how I started to learn more about system architecture.
If you're interested github.com/donnemartin/system-design-primer is a really good resource.
Thanks
me neither, lol.
done projects using postgresql for about 3 years, but haven't used any of these tools or faced problems like this due to projects being built for low user counts (local industrial facilities etc)
Read up on scaling up a database....that's the basics basically then a lot of it is system design
very beautifully made video, i wonder how it has only 2.7k views with such high quality
haha thank you! I just like making these type of videos and I just hope that my viewers learn a thing a two :)
the views are coming
Way interessting that primegean screaming like an idiot and giving hot takes on everything
As an ops Person, these software devs did a solid job of designing their infrastructure
Great content, definitely need more stories like this!
Amazing video. I appreciate every bit of effort you put into creating this video.
Thank you for taking the time to watch this video! :)
Best tech channel I've seen in a long time, great thumbails and content, you're bound to blow up! I would say to keep going on the company architecture case studies with practical takeaways, they're super insightful as a viewer. Also don't be afraid to ask people to subscribe if they enjoyed the video at the end, I think it's been proven to actually work.
I showed your comment to my gf who is my thumbnail designer and she was very happy to hear this! Thank you for making her day!
And yeah I'll definitely be giving your suggestion a try in the next video.
Please continue to do these types of videos it’s such an important insight that isn’t common on youtube
Interesting case study! Thanks for the great explanation and engaging visuals; keep it up :)
I'm glad the visuals were engaging!
Damn 🔥, you'll react 1M soon. The quality of video and explanation is quite smooth.
Just stumbled upon this channel.
This video is top tier!
---
Earned a sub
Great content, very well explained with nice animations.
Can you tell which tool you are using for this animations?
great content, we need more of this.
Thank you and more are coming!
This was great, thank you for sharing!
thank you for taking the time to watch!
Great content and nice story telling...
It’s a lot of data engineering and data infra work, which not many people appreciate. Great content.
This channel is a god send. As i am moving into more senior roles, these videos are gold mine.
That's awesome hopefully you will stick around for more videos like this!
@@kikisbytes you bet. Notifications turned on 🫡
This is GOLD, thank you for sharing!
Where do you get this information?
Thank you for watching! The original blog is in the description if you want to check it out :)
Another great video!
Glad you enjoyed it!
teaching style very cool and easy thx
Very valuable content! Love this case studies
Heyy thank you very much! Also I checked out your channel and damn some very interesting videos! How long did it take to build one of those lego cars?
@@kikisbytes Some of the projects take a lot of time. I imagine your elaborated videos also must take a lot of time and effort to complete (research, script, animations, etc)
I hope I had more time to make videos more often but at the moment I'm very busy with uni and some web dev and ml projects. I hope this summer I will have more time to release new content again.
Keep up the good work!
looking forward to seeing you publish your next video!
Great video.
My guy. Great use case. Would love to hear more abstract strategies how to solve general problems. But this was neat and awesome. Great entertainment plus could use it for my work.
Ps. : an other suggestion would be to evaluate if sharding with consistent hashing would be a good avenue to follow. Generally, I might be great solution
Great video, thank you very much. Subbed.
much appreciated!! Stay tune for more!
Using a cache to compensate for replication lag is brilliant
the sharding part is very complex topic. I have never done sharding the database. I read somewhere that sharding is probably the last option you should do when it comes to scaling the database because of how difficult it is
Hi Kiki, the video was very informative and I really enjoyed the visuals. What do you use to make them?
Motion canvas. Sorry for bothering you before checking other comments!
hahaha no worries!! And yeah I use motion canvas for the animations. I was inspired by the creator of motion canvas to start using it. His channel on youtube is `aarthificial` if you're curious.
When you upload a new image into an albums. Then immediately when you come back to the album, the new image shows up wrong. And only after the 2nd refresh, would it show the correct new image. I noticed this as of 2023/4
This was great. Could you also make a video explaining the working of sites like Canva? Thank you
great video
I have such a hard time trying to not think "OK so all they needed was two larger, but not large, AIX boxes or RAC and instead they engineered around" till they had finally cleaned up things back to a manageable design.
nice and clean presentation. What app/solution did you use for the visualization?
I did most of the animation using motion canvas!
nice,keeeeeeeeeeeeeep going 🚀🚀🚀
When you said Membase, is that the same thing as Couchbase?
Bro how do you get all this operational detail. Great stuff
The original article in the description if you want to learn more. It was quite hard to follow so that I had to spend a bit of time trying to understand what they were doing.
What tool is used to create these animations?
good job
Thank you!
Wow!
How do you know this stuff? How programs worked and how they changed and what software they used?
the original blog is in the description which I used to understand what was done and I fill in missing details as needed (how some technology work etc).
3:20 not to be confused with sharting, which is *much* less favorable.
Explained distributed databases better than my professor! Thank you
3 questions:
1. How many years of experience did your professor have in the industry?
2. Am curious, how exactly did the video explain it better?
3.. Which university was this if you don't mind sharing?
@@skyhappy1. Ten years
2. Well, now that I think about it, I think I just needed examples. He did a good job of explaining the conflicts one might have with distributed databases when there are lag/shutdowns, but he didn't cover sharding very clearly. But yeah, personally I learn a lot through examples and he didn't give many.
3. University of Manchester
@@altertopias Examples are super important, I like them as well. One of my grips with professors at my uni, uni of Toronto, is that they give textbook definitions but barely any examples. This happened in the SQL databases course, operating systems, networking, and more. I think it's a sign of poor knowledge on their part, as they are just regurgitating what the textbook says. Only someone who understands it well is able to give good examples.
Also, what year did you take the class? Wondering how recent it was.
Thanks!
Thank you for watching!
very good very nice
i love denormalization
How do you know the issues they faced with such detail?!
you can check out the original blog in the description!
Which technologies did you use to create this awesome video? haha
thank you for the compliment! I use motion canvas for the animations
How do they manage that lots of instances?
that's a good question and I don't know the answer to but I'm sure they have load balancing, service discovery, auto scaling and some monitoring to handle the scale.
Im new to web development so i dont understand anything here. What is EC2, S3, CloudFront, NGINX? What even are they used for?
Everything except nginx is a AWS Service with a fancy name, oversimplified the services are as follows:
EC2 -> A virtual private server (VPS)
S3 -> Some kind of external HDD with a fancy api
CloudFront -> A CDN, Webservers serving static files like images but from a server which is physically as close to you as possible
Nginx -> A webserver, compare it to apache2
Cool. Thank you for taking the time to respond. All of this is just so confusing.@@soundzthevibewalker
heyy first of all thank you for taking the time to watch the video!
What are your thoughts if I add some brief descriptions of these technologies in the description? Would that be helpful in future videos?
@kikisbytes Yes. That would be tremendous help. Especially for beginners like me to help us go along with your videos and not get lost.
what Pinterest whatsap and discord have to do with erlang?
"Everything will fail, so try to keep it simple." - wise engineer
yeah that's why I will integrate AI, ML, deep learning, blockchain, cryptocurrency into my todo app.
Software doesn't require that many engineers if you put quality first. Once something is done it doesn't require change unless you did something wrong.
lmao okay. I wish I had that experience.
Yep one of the takeaway that they mentioned which really resonates here
I just sharded my pants
LOOOL what. At least you can clean each part of your pants individually :P
They basically created MongoDB with MySQL
Thank you algorithm for bringing me this amazing and hidden content
awhhh 😭 thank you so much for the kind words!!!
Are they still running php?
They never ran PHP. The video mentioned the LAMP stack only as an example of what other comanies were using. They used mysql like in LAMP but they used python instead, with a mix of modified Django for their initial service and other frameworks for specific parts.
They also adopted react early to try to make the frontend avoid repeated requests to the server if it had already fetched the relevant data.
@BosonCollider they would have been better off with PHP as it's orders of magnitude faster, requires less resources, and highly scalable.
very nice video subscribed if you keep the graphics same and use the same topics like these
thank you! Def will be seeing a bunch more type of videos coming soon
After watching your other videos of companies growing to scale it seems like the engineers at pinterest always chose the most complicated solutions 😭
it's called job security 😉
sharting!
sharting
i mean, the site doesnt do anything useful so im not surprised they managed. the amount of times ive looked for an image only to find frikkin pinterests thumbnail-sized (resized) crap was what lead me to make the google chrome extension that removes all pinterest results from your google searches
that sounds interesting, is your extension open source by any chance? Would love to check it out.
@@kikisbytes Ofcause those extensions are open source just like any Javascript program. Those extensions to prorect people from malicious Pinterest's practices were available for all the browsers in various forms.
Shardt
The real answer is SEO spam and making image search unusable
What happened since 2012? :)
2:15 Alogirthm?? ALOGIRTHM??!
omfg I can't believe I missed this 🙈
so they had 40 engineers 10+ years ago, guess they have thousands now.
yeah I wouldn't be surprised if their employee number is in the thousands.
sharding not to be confused with sharting
🙈
similar concept though. Your digestive system gets over loaded and you shart out a mess.
A place with a lot of cr*p pictures ? I often have to remove pininterest results from the search to avoid irrelevant data. Regrettably, trash is getting popular these days.
haha, Alogirthm
so embarrassing haha 🙈
Only 6 engineers?
That costs a million dollars a year In the US.
Why would you need a hundred devs.
Small teams get stuff done.
Pinterest has a very specific focus and feature set, where as other platforms are often doing several dozen things, like a phone system, a Messenger system, video Infrastructure and in some cases being outright full blown ISPs
Bcz they didn’t use Rust
Rust didn't exist and would have slowed build time wayyy down. Go or PHP would have been far better than Python
bUt wHy NoT JuST haVE a MONOl1tH
I know the answer from experience: Spam. They somehow convinced Google to show them whenever I search for pictures. None of the pictures they host are available for free, so people who are desperate become customers. That's not a model I'd recommend anyone else reproduce and it's certainly nothing to admire.
Loved the videos and as a cs student, I hope to become a problem solver as these guys.
I am also interested in learning database design, would you recommend any course(preferable) or books?
thank you and you will eventually!!!
Hmm I recommend checking out:
1. Hussein Nasser's channel on youtube. He has some really cool videos on backend.
2. Alex Xu's system design books (honestly helped soo much back then)
3. Designing Data-Intensive Applications (book)
Hope these help you on your journey!
@@kikisbytesThese sound good for future but I was actually asking for some core database resources like once that teaches the core concepts like ACID principles, normalisation, etc.
I had one database book during my uni but that was dense theory and not very practical. If you have any recs for that then I would appreciate you sharing them
@@dickgrayson4237 ahh I get you. Yeah sorry I don't really have any specific course that I can recommend :( I learned about these concepts through interview prep, work and blogs that I come across.
they made a worse version of cassandra
That's nothing. Whatspp got to billions of users with just 35 employees.
That's one of the case study that I want to cover in the future :)
Pinterest loading is built like garbage, not like doomscroll addicts seem to car about their pictures taking 30 seconds to load
You kidding? It takes like a second max to load everything in every page. It just seems slower because no lazy loading elements
tl;dr they stole the internet
Pinterest is a blight on the internet. Utterly pointless website that pollutes search engines.
So sharding, eww.
pinterest is the cancer of internet and i'm baffled that it existed for so long time
Go cry that you can't do anything like it 😂
I could tell you're asian without seeing your face
It's your accent
guess you'll have to stick around to find out :p
If they started using Parquet and did not just copy and paste tech stacks it would only need 1 engineer