Razorpay's Journey to Microservices w/ Arjun | Ep 1
Вставка
- Опубліковано 23 лип 2024
- System Design for SDE-2 and above: arpitbhayani.me/masterclass
System Design for Beginners: arpitbhayani.me/sys-design
Redis Internals: arpitbhayani.me/redis
Build Your Own Redis / DNS / BitTorrent / SQLite - with CodeCrafters.
Sign up and get 40% off - app.codecrafters.io/join?via=...
In the video, I discussed the challenges of maintaining data consistency in microservices, focusing on the payment system at Razorpay. Initially, they used a monolithic architecture with MySQL for strong data consistency. However, as traffic grew, they transitioned to microservices to handle the load. They faced issues with database connection limits and decided to split core entities like payments, orders, and ledger into separate microservices. The transition was gradual, involving dual writes to ensure data consistency before fully transitioning traffic.
Recommended videos and playlists
If you liked this video, you will find the following videos and playlists helpful
System Design: • PostgreSQL connection ...
Designing Microservices: • Advantages of adopting...
Database Engineering: • How nested loop, hash,...
Concurrency In-depth: • How to write efficient...
Research paper dissections: • The Google File System...
Outage Dissections: • Dissecting GitHub Outa...
Hash Table Internals: • Internal Structure of ...
Bittorrent Internals: • Introduction to BitTor...
Things you will find amusing
Knowledge Base: arpitbhayani.me/knowledge-base
Bookshelf: arpitbhayani.me/bookshelf
Papershelf: arpitbhayani.me/papershelf
Other socials
I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff.
LinkedIn: / arpitbhayani
Twitter: / arpit_bhayani
Weekly Newsletter: arpit.substack.com
Thank you for watching and supporting! it means a ton.
I am on a mission to bring out the best engineering stories from around the world and make you all fall in
love with engineering. If you resonate with this then follow along, I always keep it no-fluff. - Наука та технологія
You should continue this podcast series. Out of all the podcasts or technical tutorials, there is no one in this niche. Students or experienced professionals will learn a lot from this. Kudos to you.
Check out other videos on my channel. You will find them interesting.
Pp0
*timestamps:*
0:00 Introduction
2:48 Initial architecture/techstack of RazorPay
4:30 ensuring strong consistency on monolithic architecture
9:06 check constraints
10:47 deciding to move from monolith to microservices
15:23 splitting of databases/application logic when moving to microservices
18:25 dual writing of data into monolith as well as microservice dbs(during transition phase), transaction outbox pattern
21:19 databases in current microservice architecture
22:45 deciding on SQL vs NOSQL
25:10 ensuring consistency in microservice architecture
28:10 payment flow with eventual consistency
34:40 prioritization for settlements@T+0
37:20 chaos testing
39:00 integration tests on interface between 2 services
41:00 incidents when things went wrong
47:40 outro
Thank you so much for this.
The way you progress the communication through passively narrating and curiously listening made this podcast different from other existing system design podcasts.
Such a technical dense podcast, rare to find such podcasts, nothing less than a technical conference talk. Great job Arpit !!
These 49.48 mins went pass like some couple of minutes.
This is so much learning and that also in such such basic terminology. That's one heck of a enginnering.
Thanks for the podcast Arpit and Arjun. I wish this series never comes to an end ❤️
Such an amazing and detailed conversation. Absolutely loved it 💯
Few things amazed me and I am not sure about whether its worth it.
1. Just for connection pooling they migrated monolith to microservice.
2. They remove foreign key constraints in their table and made custom orm.
This is one of the best Podcasts I have ever seen. Such deep and niche insights. Beauty of Engineering!
Great conversation !! Covered almost end to end high level architecture, challenges etc.
Loved every bit of it . Please continue this series .
loved this episode, so much to learn from razorpay. The data consistency section and ledger settlement is such a creative insight for anybody who hasn't worked on it yet. Thanks for recording this session and happy 1 month @google Arpit!!!
Just one word.. Brilliant Podcast
This is really Asli Engineering Stuff. Worth watching every second of this podcast, can't skip anything. Very Insightful.
Wow! what an amazing podcast. Looking forward for more such Videos.
Amazing stuff! Looking forward to more such podcasts.
Thanks for the video to both of you, the content is really phaad!!
Great podcast Arpit. Best thing I loved was that you were paraphrasing most of the things that Arjun said. I think you already know most of the things and still you sounded like a newbie with excitement who's listening about systems for the first time :)
Can’t state how insightful it was. It blew my mind. Discussion really took my understanding to next level.
Wow Arpit....very knowledgeable session, Thanks for bringing him and the insights
Thank you so muchh arpit.... It is true knowledge that will help engineers in real time, who are newly taking you challenges.
Awesome talk. Thanks for doing this.
Brilliant value addition to engineering contents, real stuff! ❤️
As always superb content and also I had recently cracked offered from Razorpay and Its Interview Process is also Amazing.
Brilliant podcast! Thank you arpit for doing this!!
You must continue this series.... Phenomenal insights... Waiting for more
woow, my mind blows away, so much knowledge on a plate, lots of things to get to know about how the actual digital payment system works, which may appear simple from a normal person's perspective. Thank you, Arpit.
loved It, amazing things are doing by Razorpay Engineers. 😍
Wow! What a knowledge packed podcast and this is just ep1! So glad this exists! Really enjoyed conversations between both of you!
Excited for future eps! Thanks Arpit!
Informative session. Learnt so much!
i am very excited for this series.
#thanks#gratefulToArpit
Great discussion. Enjoyed and got to learn new things. 🔥
Very interesting and informative. Loved it.
Pure gold content 👏
Best engineering interview out there. loved it.
Keep this series going!!! So much knowledge in one video
I have a ton of videos on various topics that you'll find more interesting than this. Scroll through my channel once.
really insightful video , keep up the good work
superb… really learned lot of things from this session.🤘🏻
Thankyou for providing this content ❤, really insightful
Really loved this one, thanks Arpit for bringing such insightful engineering podcast
Looking forward!!
Thanks for this amazing video.
Lots of engineering insights and technical points. Got to learn many new things.
This is Asli Hardcore Engineering! Thanks a lot Arpit for bringing this amazing episode. Really amazed by the engineering practices at Razorpay. The way Arjun explained each and every detail was enlightening; definitely a skill to be learned by every software engineer. It is one of the best engineering podcasts and I got to learn so much!
Thanks Arpit Bhaiya for this awesome podcast.
Really Enjoyed it
thanks
Need More Sir
Great start of the day.
Loved it.. insightful
Great video, we want more of these
Learned a lot from these 49 minutes! Hats off to the engineers!!
Mind blown podcast 🤯Thanks Arpit. You really inspires many like me. 😁
A brilliant show. Thank you guys. One thing that I am still interested to know. I have seen so many people talks about a lot of requests we were getting. But there is no number. So if you could tell this in your next video. Then it would be really great.
Really insightful and learnt a lot.
need more podcasts like this
gold content. I got to learn from this video a lot .Thanks Arpit .
So true!
This is truly asli engineering . Seedhi Baat no bakwaas, Real domain challenges and solution . I know that no architectures is The BEST. OutBox CDC ... amazing stuff ... please continue making stuff like this.
Thanks arpit such a wonderful podcast
Was waiting for this one.
Really inspiring series of videos, arpit. Keep it going I am sure it is inspiring a lot of Indian techies to dive deep and try these things on, it will definitely make better quality made in india for the world systems at an unbelievable price point.
This style of system design discussion is lit and interesting af
There are bunch of videos on my channel on real-world System Design. You will definitely find them interesting.
Great podcast very informative, thanks
maja agaya sir bas 🙏
ase podcast kar te rahiye
Great watch. Couldn't understand few of the technology used as my experience is mostly related to MS stack. But it has generated lot of curiosity about the stack used at RazorPay. Looking for more such podcasts. Thanks Arjan/Arpit for this wonderful discussion.
Very nice podcast , Enjoying the technology
Real engineering hits more than any dopamine.
Got facinated by the number of DBs razorpay is using depends on seevice.
Really cool stuff with cool Engineer
Wow great video, such complex concepts put so simple without using a single diagram.
Good going arjun…
video aane se pehle hi dekh liya tumne
@@saumitrashukla591 ji shukla ji, it was open for sometime, you got late
Thanks Varun 😊
Great video, one interesting question would’ve been what have been some issues with microservice architecture which still they’re in process of solving or need to evolve more
We really need these kind of podcasts. Thank you Arpit
Would recommend going through other videos of my channel.
Very informative content
As Always Arjun is great at explaining!
It always amazes to see real world problems.
Pure Gold
Brilliant Engineering by razorpay.
Please get more such videos
just one word: wow!
Amazing ♥
I think one more step before dual write is to backup the existing data so that traffic can be served
It’s really difficult to think that a foreign key validation at PHP (application) level would be faster than what MySQL can do. Database are optimized for these things.
not about the speed. But having the ability to split.
Curious about how uniqueness & race conditions regd database writes were achieved by moving db validations & constraints to application layers.
Would love a 20 mins video where you can make notes about this conversation and explain. There were some terms and flows which i couldnt follow. CDC was new, sounds like ddb streams. Also the part where they ensure the ledger update was pretty cool but I couldnt understand it fully.
The interviewee is providing calm and composed answers. I liked the clarity and the interviewer has to improve and avoid rephrasing every answer for such.a long time
How can you decide to which bank you should route request its chooseen by customer
Intial tech stack:Php, Larvel, Route 53, alb, mysql rds
Going good...
Good one bro
Payments data can be huge is rdbms enough to support the data , are you removing payments data periodically to some cold store from the transactional rdbms?
When we said MySQL is high on consistency. which consistency we r referring here..ACID one or CAP one.. it sound like acid one but then you removed all the constraints from db.. which is bit conflicting @arujn
Deadlock not raising exception (Causing dirty ready) should this not be happening by default in all rdbms databases? Curious is this also the case in other db's which guarantee acid?.
The last section on incidents and their resolution was good.
I just knew I too using transaction out box pattern (by name) :)
Hi Arpit & Arjun,
I've been thoroughly enjoying your content and have been binge-watching your videos.
At 18:25, Arjun discussed implementing the outbox pattern for dual writes. While I understand the concept well (and have implemented something similar), I wanted to delve deeper into the actual implementation. The traditional use case for this pattern involves achieving ACD (a form of transactionality) for database writes and event publication in the message queue or a different DB.
Once the data is transactionally written to both the monolith table and the outbox table, an asynchronous job, using transactional log tailing (binlog consumption), allows us to consistently write to the microservice table.
However, let's consider a scenario where the broker is down (acting as a SPoF). How does the OUTBOX PATTERN ensure consistency in this situation?
In my implementation, instead of relying on binlog consumption, I've set up a cronjob (using a serverless service) to read from the outbox table and publish the event to the message queue. If it succeeds, it updates the status in the outbox table to prevent the next run of the cronjob from picking it up again.
I'd love to hear your thoughts.
Hi Arpit, top video with some great insights. One request though - could you add timestamps to the video covering different topics that were discussed in the podcast, would be a huge help?
I have observed that people skip most of the part when I add timestamps.
If they jump forward and there comes a point where some context is needed, people are more likely to drop at that point.
Hence I have stopped adding timestamps.
after learning DBMS and MySQL able to understand this podcast
Instead of outboxer pattern cant we use read replica
You can add visual explanations of some technical concepts as every one might not understand/know all the concepts.
Hey Arpit,
Do read LedgerStore paper by Uber. With that in mind, you will love the beauty of both razorpay and Ubers ledger system, especially the way uber migrated their db from aws to in house and writing to both db parallels to test thing out.
Thanks a ton for sharing. Definitely going through it.
@@AsliEngineering
Please do make video on same if possible. I think it is good enough for everyone to learn something from a large db migration.
Love your content and open for any help you need (not for technical perspective as I am still learning and growing but any other task or any initiative) in growing your community.
I have always wondered about this scenario with the foreign key/constraint validation being done in application code because I have seen a couple of threads where big companies don't use foreign key constraints. Consider a situation where you are persisting entity X (which must have a unique name) and it also holds a bunch of foreign refs(say Y,Z,..). Won't performing the validation in the application code involve querying the entire X table with the name projection (check unique), querying the Y,Z,.. other tables by ID? Why is this a better option than letting the relational database handle this ? As far as I understand it, this approach will make you perform more queries than required. What if there are millions of rows (say users table) in each of the tables ?
the orm does this only for data which they wouldn't have to query for.
Hi Arpit! Eager to listen to Dr. Kailash of Zerodha in such podcasts as well. Thanks!
Tazorpay walo ne abhi naya merchant ko onboard nhi kar raha hai 😭😭😭😭Muze phonepe or paym pg jaana pad raha hai.
Amazing discussion! :D
Being a part of debugging through the last issue mentioned would be lit af :D
Yeah. Even I got engrossed and tried to get max info on it 😀
Loved it Arpit!
Good one Arpit and Arjun learned a lot, I am so stuck with AWS that can only relate to amazon specific service be it queue or notification topics, this is triggering an alarm for me to look into other similar products used at scale worldwide.
PS:~ Not sure their UTs are with 95% code coverage; hahaha Arjun can reply and confirm.
No bullshit vid. Loved it. I work as part of Platform too, can relate.
One doubt though,
Wouldn't ensuring fkey constraint at app layer be slower than having db handle it?
You have to do it when you are sharding. No way out.