By listening to it, I got to know that I also faced similar issues and worked on similar set of solutions except materialised views and redis. Happy to see myself on same path...
Sensible defaults go a long way, but they only work in certain scenarios. I commend their choice of sticking to the basics. To be honest, their main engineering efforts are focused on rewriting their codebase, and I believe that is their core competency. I wish more companies allowed frequent rewrites [where needed].
The rewrites are not unwarranted, they hit scaling requirements and then improve. Their PDF sending blog is a great example. Some engineers would probably start at that, but that's not how things work in real life. Most startups face this problem as well. Starts with java or rails and then splits into microservicrs to scale independently
No doubt this is a gem talk.. Though I feel slides could be done much better with the kind of content talk had.. Still learned a lot & cheers to this great talk!!
At 37:27, not just the speed of computation, but also the egress cost of transferring all that unnecessary data over the wire to your server from database to perform reduce type computations which can simply be done on the database layer itself. And if you find yourself querying such data frequently, then you can have materialised views for it, making it even faster. Finally, if your use case doesn't require the most latest data and is fine with eventual consistency, then you can add a caching layer in front of it, making it blazing fast.
It all felt like an interesting discussion, but I couldn't understand (won't say I couldn't digest) a few things he mentioned such as - copying terabytes of data and restarting the server under 1 min, letting the database do most of the calculative work, no replica sets (how about contingency planning) and in a way portraying Postgres as an 'all weather database'......seems like a wake up call for myself.....
Ahh TBs of data with 16 core 32 GB setup!! Me looking back at my days at startup where a 300 GB DB was running on a Out of support PSQL with 16 core 32 GB server. Literally any moderate bulk record operation would light up all 16 cores to 99.9%!! I even increased PG max number of connections after modifying Linux sysctl defaults!! Since developers were so adamant on writing everything Sync and poetic queries (it gets creative as they type). No one other than me suspected these Sync APIs, bad queries and garbage tables. No wonder my questions were not answered in DBStackExchange but all were double checking the DB size and Spec again and again 😅 Note: I was not a DBA, may be thats why our Eng team never took my points seriously. Thankful to all those learning experience though. Here I am hearing valuable points from experts.
Only hardcore techies or IT professionals will find this video a Gem💎 . Well in the era of the Graph database, MongoDB, NOSQL etc. the legend in the video is talking basically about SQL database.😂 ❤ Yes he is definitely correct. With Oracle as an enterprise paid database, we have the king of open source database called Postgres, a boon for startups. I am really surprised, zerodha with millions of customers uses Postgres..... ❤ I thought they were using some high tech AWS virtual nosql databases 😂
It's fifteen past seven in the morning. Just found this video, have been following zeroda's journey for quite a long time. The purpose of this comment is, tried to find a single video where a production grade live application auth has shared their journey on postgresql. And finally found it. Left job, it's been 8 months. Single handedly writing the db [pg] for my dream project. All the insights shared in this video, I don't know how much thankful I am for this. The journey is too relatable! Kuddos to the team.
I don't agree with offloading all of the computations to the database. Yes some computations are good on the database side, but if you are doing some heavy computation that should be on the app side. The benefit of that is you can have multiple backend machines and single database. Using loadbalancer and multiple workers is very easy for backend as compared to having multiple databases.
Agree, suffered a lot with my analytics platform (mongo) only because we offloaded queries to db. This is even after good amount of denormalisation, huge data means aggregation are consume all the resources.
I agree with the approach where some computation could offload outside the database. The only downside is developer friction, earlier one was able to write queries directly in SQL. Whereas now one has to take care of doing it outside the database in some other way. Luckily new age tools like DuckDB help us do that effortlessly.
single db to store all the transactions placed on kite in real-time (easily 1M/Sec)... this is possible only if you have extremely high end machine and there is no fault-tolerance in this. If ur db goes down, it would take a few mins to bring it up. It's also too operational haeavy in nature..every night switching disks (though automated) ...seems like a college project. singe point of failure, heavy on ops :(
One doubt here, Redis is a NOSQL database. And from the lecture i get that Redis is used as the caching layer. And when Queries hit from Sqljobber, it checkes whether it is in the caching layer or not. If not, it goes to the primary DB (PostgreSQL). How are the queries checked between NOSQL and SQL databases?
From what I understood, what he said is They have one primary postgres db and another postgres db used as a cacheing layer or db. When a query executs, it looks on this cacheing postgres db and if data is not found there then it looks on the primary db. But they do use redis for other services cacheing.
Do they implement CQRS ?? not clear ; what kind of postgres scaling ?; Any SOR layer - System of record; what ORM ??; some high level banter about postgres DB tuning with no specifics; Query optimization no specicifs
That one slide of how do we manage big data pretty much throws light on points where MongoDB shines. Maybe they had a poor experience and wrote it off.
I guess nosql ism't really suited for a very structured thing like a stock broker database. Others he mentioned like Mysql, Cockroachdb are SQL. It's kind of just like an experimental thing. But maybe he just remembered something funny about it.
22:00 15tb of S3 file loaded into PG in 2mins !! Don't lie. U don't have slave/replicas. U don't load balance. U have a PG on top of another PG as caching? Why man? Why can't you use memcache or redis for this? These will be much faster. There is scope for a lot more improvement in the architecture.
Sorry you are not Robinhood of India in trading. Robinhood offers research and lot of reasearch and recommendations with lot of features. But you are one of best discount broker to save money of swing traders
DB backup, restoring everyday. postgres as caching layer - it seems like they just want to rely on one thing only which is absurd. Seems like they are trying to find solution to problem which shouldn't have been problem as such. Totally absurd, they are ok with everything running, donot want to do things at higher scale. How can you be ok with such kind of system.
yes, because he mentioned that they had made this changes by looking on the specific data that are dealing with and anybody else doing the same thing with there data won't work the same
Terabytes of data copied to a server and starting in less than a minute? Are you joking? Copying the data from S3 to your local server itself will take the time for terabytes of data. May be I am missing something here.
Now you know reason why zerodha is down every now and then. Haha.. but on lighter note i think for trading broker they should have robust backup mechanism in place, s3 to db in minute is a dream scenario but that works for them so who knows
100% not possible even with newer instances with more than 100Gb of network speed. The ones with more than 200Gb of network speed requires special drivers and interfaces. Either he was bluffing or misremembering details. As they were partitioning by month, it could be possible that he is referring to restoring each DB hosting a month/s worth of data in under 2 minutes as that is possible. But data in that single server would be in the range of 1 or 2 TB and even then only with compression it is possible to do it under a minute.
Zerodha doesn't have replicas and doesn't have slaves. All they have is a sorry when their server or db crashes. People lose money, and they think replicas are not important.
100% not possible even with newer instances with more than 100Gb of network speed. either he was bluffing or misremembering details. As they were partitioning by month, it could be possible that he is referring to restoring each DB hosting a month/s worth of data in under 2 minutes as that is possible. But data in that single server would be in the range of 1 or 2 TB only.
Can share RCA for Issue we are facing since last two months. I am Software engineer and also Customer of Zerodha. I want to know why other broker are runing fine and zerodha not.
People say c++ is retarded/obsolete. I dont agree. How you can achieve super-performace, responsiveness, better synchronous result without using a natively driven code.
What does he mean by checkpoints of data backup? As in they dont take entire backup from the start to current? Just a fixed period of time like a month?
overloading postgres is not helpful and dont do it specially when you have no concept of nodes . postgres servers are not free even. if you are not able to write code for sum/avg for close to same speed as postgres service then you need better engineering. Architectures become complicated when you have distribution/SOA . Either this is half knowledge served or this platform is running on hopes and dreams. And also S3 is not that fast
This is feels like a 3rd year B-tech class, not a techie presentation. The presented points can be found across multiple b-tech computer science books. May be he is not a tech person thats why he didnt gave the examples on tuning.
wow, such an unconventional video & breaks a lot of myths, power to figure out stuff yourself and not do what everyone else is doing. but that being said, we will also have to understand that there have been outrages from Zerodha and people have lost money, i am not saying it could. be because of this very unconventional postgres setup but who knows, zerodah def doesn't have 99.99 uptime
![Hands-On Power BI Tutorial 📊Beginner to Pro [Full Course] ⚡](http://i.ytimg.com/vi/5X5LWcLtkzg/mqdefault.jpg)
Boring tech is usually better in most cases.
These are the guys and their content which people should be consuming and not those of LinkedIn didis
You hit the nail on the head!
No Keerti was harmed by this comment.
By listening to it, I got to know that I also faced similar issues and worked on similar set of solutions except materialised views and redis. Happy to see myself on same path...
Zerodha sits on top of SEBI's tech APIs. God, One has to listen from SEBI on how their orchestration is.
Not SEBI, NSE and BSE
@@mrinalkamboj3196 Yes right.
Sensible defaults go a long way, but they only work in certain scenarios. I commend their choice of sticking to the basics.
To be honest, their main engineering efforts are focused on rewriting their codebase, and I believe that is their core competency. I wish more companies allowed frequent rewrites [where needed].
The rewrites are not unwarranted, they hit scaling requirements and then improve. Their PDF sending blog is a great example. Some engineers would probably start at that, but that's not how things work in real life. Most startups face this problem as well. Starts with java or rails and then splits into microservicrs to scale independently
PG always shock engineers with new features and flexibility
Great talk. So refreshing to see someone talk about actually getting stuff done instead of just some over-engineered setup
I love this guy seriously. Very powerful insights.
No doubt this is a gem talk..
Though I feel slides could be done much better with the kind of content talk had.. Still learned a lot & cheers to this great talk!!
At 37:27, not just the speed of computation, but also the egress cost of transferring all that unnecessary data over the wire to your server from database to perform reduce type computations which can simply be done on the database layer itself.
And if you find yourself querying such data frequently, then you can have materialised views for it, making it even faster. Finally, if your use case doesn't require the most latest data and is fine with eventual consistency, then you can add a caching layer in front of it, making it blazing fast.
It all felt like an interesting discussion, but I couldn't understand (won't say I couldn't digest) a few things he mentioned such as - copying terabytes of data and restarting the server under 1 min, letting the database do most of the calculative work, no replica sets (how about contingency planning) and in a way portraying Postgres as an 'all weather database'......seems like a wake up call for myself.....
Ahh TBs of data with 16 core 32 GB setup!! Me looking back at my days at startup where a 300 GB DB was running on a Out of support PSQL with 16 core 32 GB server. Literally any moderate bulk record operation would light up all 16 cores to 99.9%!! I even increased PG max number of connections after modifying Linux sysctl defaults!!
Since developers were so adamant on writing everything Sync and poetic queries (it gets creative as they type). No one other than me suspected these Sync APIs, bad queries and garbage tables. No wonder my questions were not answered in DBStackExchange but all were double checking the DB size and Spec again and again 😅
Note: I was not a DBA, may be thats why our Eng team never took my points seriously. Thankful to all those learning experience though. Here I am hearing valuable points from experts.
So finally found the responsible guy for the recent backend glitch..
video full of real time challenges and the power of postgres
Only hardcore techies or IT professionals will find this video a Gem💎 .
Well in the era of the Graph database, MongoDB, NOSQL etc. the legend in the video is talking basically about SQL database.😂 ❤
Yes he is definitely correct. With Oracle as an enterprise paid database, we have the king of open source database called Postgres, a boon for startups.
I am really surprised, zerodha with millions of customers uses Postgres..... ❤
I thought they were using some high tech AWS virtual nosql databases 😂
Thanks for the presentation. Appreciate it
What about durability? what if the db instance goes down? He mentioned they use a single node db without replication. How does that work?
It's fifteen past seven in the morning. Just found this video, have been following zeroda's journey for quite a long time.
The purpose of this comment is, tried to find a single video where a production grade live application auth has shared their journey on postgresql. And finally found it.
Left job, it's been 8 months. Single handedly writing the db [pg] for my dream project. All the insights shared in this video, I don't know how much thankful I am for this.
The journey is too relatable! Kuddos to the team.
craazzy setup man 25:00 so minimal and simple.
I don't agree with offloading all of the computations to the database. Yes some computations are good on the database side, but if you are doing some heavy computation that should be on the app side. The benefit of that is you can have multiple backend machines and single database. Using loadbalancer and multiple workers is very easy for backend as compared to having multiple databases.
Agree, suffered a lot with my analytics platform (mongo) only because we offloaded queries to db. This is even after good amount of denormalisation, huge data means aggregation are consume all the resources.
I agree with the approach where some computation could offload outside the database.
The only downside is developer friction, earlier one was able to write queries directly in SQL.
Whereas now one has to take care of doing it outside the database in some other way.
Luckily new age tools like DuckDB help us do that effortlessly.
single db to store all the transactions placed on kite in real-time (easily 1M/Sec)... this is possible only if you have extremely high end machine and there is no fault-tolerance in this.
If ur db goes down, it would take a few mins to bring it up.
It's also too operational haeavy in nature..every night switching disks (though automated) ...seems like a college project.
singe point of failure, heavy on ops :(
Very good open talk by the zerodha presenter...
Awesome!!
looks at his eyes... so much thinking going on..
frequent code rewrites === frequent down times (if the code changes do not have sufficient coverage and are not properly tested by QA)
hahaha ..good one! Very valid points were made!! x)
Oracle u didn't mentioned, pg is open source that's why used
Hope Kamath brothers are paying you well..
Super interesting :) more such insights tech talks🙌
am i missing something here? Able to bring up db 15-20 TB of fresh copy from S3 in < 2 mins!
yes he said that
Is it true
@@namestelearning no idea
I mean how
not sure, is it possible with the largest ec2 instance and with largest disk iops?
I have always found simple dumb solutions to be the best solution that lasts long without any crash.
One doubt here, Redis is a NOSQL database. And from the lecture i get that Redis is used as the caching layer. And when Queries hit from Sqljobber, it checkes whether it is in the caching layer or not. If not, it goes to the primary DB (PostgreSQL).
How are the queries checked between NOSQL and SQL databases?
From what I understood, what he said is
They have one primary postgres db and another postgres db used as a cacheing layer or db.
When a query executs, it looks on this cacheing postgres db and if data is not found there then it looks on the primary db.
But they do use redis for other services cacheing.
Understanding this is as exciting as watching paint dry.
What's on the sealing? Still learned a lot & cheers to this great talk!!
Boring is good for you. Thanks for sharing the knowledge.
Great talk , got to learn a lot from this video 😍
Do they implement CQRS ?? not clear ; what kind of postgres scaling ?; Any SOR layer - System of record; what ORM ??; some high level banter about postgres DB tuning with no specifics; Query optimization no specicifs
Do they invest via Zerodha?
He wants to keep talking light but the audience is damn serious
Atleast mention the speaker name in video title or in the description
I feel dumb
1:42 why there is small laugh on mongodb 😂
Yeah WTF
That one slide of how do we manage big data pretty much throws light on points where MongoDB shines. Maybe they had a poor experience and wrote it off.
I guess nosql ism't really suited for a very structured thing like a stock broker database. Others he mentioned like Mysql, Cockroachdb are SQL. It's kind of just like an experimental thing. But maybe he just remembered something funny about it.
Hahaha same feeling
Where does this kind of tech conferences happens?
this is gold !☺
This is good, are these conferences invite only?
That's good to know😊
Thank you for the video....
Can you share the RCAs on recent failures on the kite platform
22:00 15tb of S3 file loaded into PG in 2mins !! Don't lie.
U don't have slave/replicas.
U don't load balance. U have a PG on top of another PG as caching? Why man? Why can't you use memcache or redis for this? These will be much faster.
There is scope for a lot more improvement in the architecture.
Sorry you are not Robinhood of India in trading. Robinhood offers research and lot of reasearch and recommendations with lot of features. But you are one of best discount broker to save money of swing traders
It was a simple comparison for understanding the product and its domain. Its not an official statement of the company
DB backup, restoring everyday. postgres as caching layer - it seems like they just want to rely on one thing only which is absurd. Seems like they are trying to find solution to problem which shouldn't have been problem as such. Totally absurd, they are ok with everything running, donot want to do things at higher scale. How can you be ok with such kind of system.
Wow
Awesome
so informative
Good lecture ⭐
Great!
why most of the crucial times zerodha app stock responding?
is it okay to share your applications db architecture like this?
yes, because he mentioned that they had made this changes by looking on the specific data that are dealing with and anybody else doing the same thing with there data won't work the same
@@BashDroid was this organized by zerodha or postgres?
none, it's done by the postgres community in india.. pgconf
why that smirk at mongodb 😢
gold
ha bhai?
@@go1d
Terabytes of data copied to a server and starting in less than a minute? Are you joking? Copying the data from S3 to your local server itself will take the time for terabytes of data. May be I am missing something here.
Now you know reason why zerodha is down every now and then. Haha.. but on lighter note i think for trading broker they should have robust backup mechanism in place, s3 to db in minute is a dream scenario but that works for them so who knows
I think they're not compressing the data. But that is way too wasteful.
they must be loading that 500 GB of cache data or max 1TB.
100% not possible even with newer instances with more than 100Gb of network speed. The ones with more than 200Gb of network speed requires special drivers and interfaces. Either he was bluffing or misremembering details. As they were partitioning by month, it could be possible that he is referring to restoring each DB hosting a month/s worth of data in under 2 minutes as that is possible. But data in that single server would be in the range of 1 or 2 TB and even then only with compression it is possible to do it under a minute.
Zerodha doesn't have replicas and doesn't have slaves. All they have is a sorry when their server or db crashes. People lose money, and they think replicas are not important.
Better if you can move to the cloud. AWS or Azure both provide great support.
What is his name?
No replica? So what happens if the DB crashes and data gets corrupted? You loose customer data!
time to switch to another broker ? :P
15tb in 2 mins !!! Really?? How ??
100% not possible even with newer instances with more than 100Gb of network speed. either he was bluffing or misremembering details. As they were partitioning by month, it could be possible that he is referring to restoring each DB hosting a month/s worth of data in under 2 minutes as that is possible. But data in that single server would be in the range of 1 or 2 TB only.
When did he say that?
Wondering why Zerodha crashes!
What is his name
Can share RCA for Issue we are facing since last two months. I am Software engineer and also Customer of Zerodha. I want to know why other broker are runing fine and zerodha not.
no you are not, because if you were you would've asked this question from your team.
The way he laughed at MongoDB😂
What is the name of the person speaking?
A little bit over provisioning when serving 2 million requests with an sla is better than being conservative.
Caching is the answer to all of your questions in this talk 😂😂
Caching might be a very bad idea in case of Trading apps.
@@tanaygupta2009 i may be agree but how
too much tuning these guys are doing that their app having lot of glitches every now and then
People say c++ is retarded/obsolete. I dont agree. How you can achieve super-performace, responsiveness, better synchronous result without using a natively driven code.
What does he mean by checkpoints of data backup? As in they dont take entire backup from the start to current? Just a fixed period of time like a month?
It means a delta load with a specific start point instead of a full load
damnn
31:19 Postgres Learnings
Bruh he laughed when he said mongo
I want to connect with you? What is your name?
now we understood why glitches are happening frequently in Zerodha..
overloading postgres is not helpful and dont do it specially when you have no concept of nodes . postgres servers are not free even. if you are not able to write code for sum/avg for close to same speed as postgres service then you need better engineering. Architectures become complicated when you have distribution/SOA . Either this is half knowledge served or this platform is running on hopes and dreams. And also S3 is not that fast
This is feels like a 3rd year B-tech class, not a techie presentation. The presented points can be found across multiple b-tech computer science books. May be he is not a tech person thats why he didnt gave the examples on tuning.
It works... until it doesn't
Just out of ignorance.. I think snow flake willl be a good choice? Can anyone comment
I think it is no longer the largest stock broker. Groww over took it since oct 2023
Groww is stupid, just because of marketing it is gaining market share
@@theshashank11 But that doesn't change the fact that they are the largest at present.
Ku6 bhi?? Pehle jo loss karwaye ho uske bare me video nikalo.
i watched this video in 1.5x still felt slow and boring.
So much talk yet no stable apps😂
Aaaaa.....aaaaaa.....aaaaaa......aaaaaaa..
Amm emmm aaaaaa aaaaaa... What that.... Silly
-rm -rf the entire data >_
wow, such an unconventional video & breaks a lot of myths, power to figure out stuff yourself and not do what everyone else is doing.
but that being said, we will also have to understand that there have been outrages from Zerodha and people have lost money, i am not saying it could. be because of this very unconventional postgres setup but who knows, zerodah def doesn't have 99.99 uptime
Did someone punch you in your chest before you spoke?!
Awkward!
What's his name ?