Hi Keerti and Gaurav, The way through which the data is communicated to the brokers via Exchange is way too much efficient and the assumptions that are taken are far off from the real values. 1. The number of stocks on NSE + BSE is around 7.5k, but apart from that there are derivatives of the stocks and indexes (NIFTY and BANKNIFTY) the most popular ones. These derivatives are the most traded ones on the exchanges. If you include these derivatives as well, the number goes beyond 1Lakh. There are times where the number of ticks (Liquid + Illiquid) goes beyond 1 lakh ticks (websocket ticks) per second. The market goes for around 6.5hours. That's humungous. 2. The exchanges opens up UDP port for multicasting. You have to listen to these multicast ports to get the information. Now that information is not a simple JSON payload or an array of numbers. It will be very inefficient and costly. It is in a compressed format that too using a very old compression algorithm as it was designed years ago and it is still being used. I have left out way too many things and this is just for listening the feeds by the broker. The real challenge comes when you listen to these feeds and distribute it towards your customer base which is growing. So scaling comes next. I would recommend going once through the documentation of packet - archives.nseindia.com/content/press/Realtime_FAO_L1.pdf This is just for educational purposes. :) A genuine feedback, The broker in itself is a very complex design, each feature of it actually is complex. What you could have done is taken one feature of it and then end up explaining it, in this way you could have been able to do more research on it. Just my honest feedback, nothing else. Kudos to both of you.
I dont think we need to keep track of all watchlist for each user on server-side. - Rather, we can simply keep watchlist info on client side(like watchList1:{stock1, stock3, ...}). - In backend, WatchlistService can just keep a mapping of user to stocks he is interested in(across all his watchlists) and send the user this group of stock entries every second(or whatever freq we decide). - Then, the client can do the job of segregating these stocks into separate watchlists.
Hey Gaurav and Keerti, There are a few questions that I feel are not covered in this design: 1. How are stream stock prices pushed to User Client Devices (Mobile / Desktop)? A normal thought that comes to our mind is the WebSocket connection (wss://), but thousands of stocks and investors could be watching prices in the watch list and portfolio (stock price change per second for hundreds of stocks). How do you think all these stock prices are pushed to user devices? 2. While ordering for a stock on a market price, the stockbroker generally shows a price while ordering the stock and there is a possibility that before even communicating to the exchange or a network prices are changed in this way maximum orders will get cancelled, but it doesn't happen like that. So what could a mechanism or strategy to execute an order? There are a lot of hopes between receiving the price from the exchange and making an order and executing it. So how this volatility is getting handled? Think it in on a high scale.
Sorry if I missed , here are you using SQL or NO SQL for translation data, apart from catching service.SQL might will give performances issue. Also for inter service communication is REST API used , might gRPC is give more performance. Just my thoughts. 😊
Hey Gaurav, you really need a trader in your discussion. There are many wrong assumptions in terms of watchlists and users and order types and whatnot in this discussion. There is quite a mismatch between the reality of order types limit, Market, and GTT and what you are understanding. A limit order is not trigger-based. It is placed immediately to the exchange but at a given price. Now if the exchange finds a counterparty who agrees to that price, your order is fulfilled. For market order no price is given to exchange, stock is bought or sold at any available price. GTT is actually a trigger based where the broker triggers the order at a given price when the price of a stock crosses a certain threshold. Let me know if you want to rerecord another session, would be happy to be part of it or give you pointers offline as well. I am a trader as well as a technical guy who understands and is interested in system design. As always, you are quite fantastic guys and a good effort.
I think @Gaurav is correct. For Watchlist - Prices are eventual consistent like watching mactch on hotstar i.e. near real time. Great Discussion. I wish I would have view this video before one of my interview would have not making mess in interview. I have used SSE for Stcok Exchange communication, Socket for Human User/Client to Zerodha communication. I did FR,NFR,Capacity Estimation, API design, after 25 min I was clue less what to discuss.. Well done @Keerti for deep dive. Got idea how to deep dive into components.
10 K for stocks.. well there are options and futures at different strike prices and the symbol list goes in 100K nearly. and broker needs to send this to ~1 crore people at the same time.. along with past data once .. to plot the chart. Estimation needs to be done that way.
You should consider total number of active users instead no of brokers while doing capacity estimation because multiple users will be requesting for the tick prices from multiple brokers at the same time. So it will be 10k*10bytes*1k(broker)*(1M active users)
Great video! Loved Keerti's presentation. She managed to keep things simple and visualize her thoughts making it easier to understand. Lover Gaurav's inputs and facts too! 🙂 Thanks for this
1. You have to decide on the roles between Zerodha and NSE. Is GTT reposnibility of Zerodha or NSE? 2. NSE does not have to maintain 10M watchlists, Zerodha has to. Zerodha will get all stock ticks from NSE in 1 request and then serve it to their clients as per their watchlist. Zerodha will not make 10M calls to NSE every day.
I actually enjoyed this discussion more than most cut and dried talks on system design. This is closer to a real team discussion or interview. Structuring is more of a hindsight thing 😅
@@gkcs Hey Gaurav, I also feel that most of the design videos on your channel lack structure. Agreed, that this is how a real system design discussion occurs. But in interviews, I feel having a structure/order becomes important. Also, do you suggest going with an unstructured approach during interviews or should we follow some sort of an order like jotting down requirements first, then estimates, then data model and finally the meat of the services, etc.
Pull based service for limit orders not going to fly. AFAIk Brokers OMS push the pending orders at day start and exchange push event if there is any change in orders.
Signal R is configurable and has a default fall back mechanism to fall with the precedence like web sockets -> sse -> long polling, serves all the purposed at once
Explained as simple as possible by @KeertiPurswani, also the discussion is informative. And the need SQL and No-sql. Listening to you guys is really upgrading the knowledge, getting used to system design 😀😀😁. Also the need of TCP ack, ans importance of Validation services.
I was asked the same question in Datbricks interview, hope I had landed here before.BTW I had given the polling based solution which the interviewer did'nt agree
Before 22:00 when she was saying just to fetch data for wishlist, I got confused, because end user will have wishlist but she was explaining connection between broker and exchange. But then Gaurav you point it and got that cleared at 22:10
Why do we need cache for watchlist? Prices of the stocks which are in Watchlist will frequently be updated (every second) . Is it correct or am I missing something
We can cache the stocks in the watchlist. Their prices will change but the stocks will change only once in a while. Watchlists will be viewed frequently, so having the stocks cached makes sense.
@@ITTIsiKhushi-keertipurswani Oh, now I got it. Ticker names could be cached and getting the name of the ticker( to fetch the prices from separate api) from cache will be faster compared to fetching the tickers name from DB. Understood. Thank You 🤗
@@KeertiPurswani what exactly does once in a while mean? Lets say some traders update their watchlist every day, will it still be feasible to maintain watchlist in cache?
Thank you for having me Gaurav! It’s always fun to design with you! 😇
Hi Keerti and Gaurav,
The way through which the data is communicated to the brokers via Exchange is way too much efficient and the assumptions that are taken are far off from the real values.
1. The number of stocks on NSE + BSE is around 7.5k, but apart from that there are derivatives of the stocks and indexes (NIFTY and BANKNIFTY) the most popular ones. These derivatives are the most traded ones on the exchanges. If you include these derivatives as well, the number goes beyond 1Lakh. There are times where the number of ticks (Liquid + Illiquid) goes beyond 1 lakh ticks (websocket ticks) per second. The market goes for around 6.5hours. That's humungous.
2. The exchanges opens up UDP port for multicasting. You have to listen to these multicast ports to get the information. Now that information is not a simple JSON payload or an array of numbers. It will be very inefficient and costly. It is in a compressed format that too using a very old compression algorithm as it was designed years ago and it is still being used.
I have left out way too many things and this is just for listening the feeds by the broker. The real challenge comes when you listen to these feeds and distribute it towards your customer base which is growing. So scaling comes next.
I would recommend going once through the documentation of packet -
archives.nseindia.com/content/press/Realtime_FAO_L1.pdf
This is just for educational purposes. :)
A genuine feedback, The broker in itself is a very complex design, each feature of it actually is complex. What you could have done is taken one feature of it and then end up explaining it, in this way you could have been able to do more research on it. Just my honest feedback, nothing else.
Kudos to both of you.
Thank you!
This format of system design interviews which is a discussion is so so much more beneficial than a monologue. Thanks!
@@gkcs I have worked on similar System NSE send its data in udp form, most of the broker uses Bloomberg or thomson reuters sdk to get market data
I dont think we need to keep track of all watchlist for each user on server-side.
- Rather, we can simply keep watchlist info on client side(like watchList1:{stock1, stock3, ...}).
- In backend, WatchlistService can just keep a mapping of user to stocks he is interested in(across all his watchlists) and send the user this group of stock entries every second(or whatever freq we decide).
- Then, the client can do the job of segregating these stocks into separate watchlists.
@@marcusaureliusfanboy if we use client side storage the watchlist which we create in web may not available in mobile and vice versa
The exchanges usually send out feeds with prices and quantities. The broker needs to to be in multicast group where they receive this information.
Hey Gaurav and Keerti,
There are a few questions that I feel are not covered in this design:
1. How are stream stock prices pushed to User Client Devices (Mobile / Desktop)? A normal thought that comes to our mind is the WebSocket connection (wss://), but thousands of stocks and investors could be watching prices in the watch list and portfolio (stock price change per second for hundreds of stocks). How do you think all these stock prices are pushed to user devices?
2. While ordering for a stock on a market price, the stockbroker generally shows a price while ordering the stock and there is a possibility that before even communicating to the exchange or a network prices are changed in this way maximum orders will get cancelled, but it doesn't happen like that. So what could a mechanism or strategy to execute an order? There are a lot of hopes between receiving the price from the exchange and making an order and executing it. So how this volatility is getting handled?
Think it in on a high scale.
Sorry if I missed , here are you using SQL or NO SQL for translation data, apart from catching service.SQL might will give performances issue. Also for inter service communication is REST API used , might gRPC is give more performance. Just my thoughts. 😊
Hey Gaurav, you really need a trader in your discussion. There are many wrong assumptions in terms of watchlists and users and order types and whatnot in this discussion. There is quite a mismatch between the reality of order types limit, Market, and GTT and what you are understanding. A limit order is not trigger-based. It is placed immediately to the exchange but at a given price. Now if the exchange finds a counterparty who agrees to that price, your order is fulfilled. For market order no price is given to exchange, stock is bought or sold at any available price. GTT is actually a trigger based where the broker triggers the order at a given price when the price of a stock crosses a certain threshold. Let me know if you want to rerecord another session, would be happy to be part of it or give you pointers offline as well. I am a trader as well as a technical guy who understands and is interested in system design. As always, you are quite fantastic guys and a good effort.
Thanks RK, where can I reach you?
I think @Gaurav is correct. For Watchlist - Prices are eventual consistent like watching mactch on hotstar i.e. near real time. Great Discussion. I wish I would have view this video before one of my interview would have not making mess in interview.
I have used SSE for Stcok Exchange communication, Socket for Human User/Client to Zerodha communication.
I did FR,NFR,Capacity Estimation, API design, after 25 min I was clue less what to discuss.. Well done @Keerti for deep dive. Got idea how to deep dive into components.
10 K for stocks.. well there are options and futures at different strike prices and the symbol list goes in 100K nearly. and broker needs to send this to ~1 crore people at the same time.. along with past data once .. to plot the chart.
Estimation needs to be done that way.
You should consider total number of active users instead no of brokers while doing capacity estimation because multiple users will be requesting for the tick prices from multiple brokers at the same time. So it will be 10k*10bytes*1k(broker)*(1M active users)
Great video! Loved Keerti's presentation. She managed to keep things simple and visualize her thoughts making it easier to understand.
Lover Gaurav's inputs and facts too! 🙂 Thanks for this
Thank you so much! 😇
Thank you so much Gaurav and Keerti, great information.
Thank you!
This was beautiful!!!!
Thanks Gaurav and Kirti!
order tracker can be places between the price tracker and order exchange.
1. You have to decide on the roles between Zerodha and NSE. Is GTT reposnibility of Zerodha or NSE?
2. NSE does not have to maintain 10M watchlists, Zerodha has to. Zerodha will get all stock ticks from NSE in 1 request and then serve it to their clients as per their watchlist. Zerodha will not make 10M calls to NSE every day.
In my view if compared to other videos of Gaurav this one didn't have any structure and wasn't well curated it seems.
Hey, thanks for the feedback, could you share how this be more structured? Will help us improve next time! 😇
I actually enjoyed this discussion more than most cut and dried talks on system design.
This is closer to a real team discussion or interview. Structuring is more of a hindsight thing 😅
@@gkcs
Hey Gaurav, I also feel that most of the design videos on your channel lack structure. Agreed, that this is how a real system design discussion occurs. But in interviews, I feel having a structure/order becomes important.
Also, do you suggest going with an unstructured approach during interviews or should we follow some sort of an order like jotting down requirements first, then estimates, then data model and finally the meat of the services, etc.
Pull based service for limit orders not going to fly. AFAIk Brokers OMS push the pending orders at day start and exchange push event if there is any change in orders.
Great video. Looks like lot of things to learn not only in system design, but in the stock market as well :) !!
Her handwriting with a stylus is pretty great tbh, lol
Signal R is configurable and has a default fall back mechanism to fall with the precedence like web sockets -> sse -> long polling, serves all the purposed at once
Explained as simple as possible by @KeertiPurswani, also the discussion is informative. And the need SQL and No-sql. Listening to you guys is really upgrading the knowledge, getting used to system design 😀😀😁. Also the need of TCP ack, ans importance of
Validation services.
I was asked the same question in Datbricks interview, hope I had landed here before.BTW I had given the polling based solution which the interviewer did'nt agree
Before 22:00 when she was saying just to fetch data for wishlist, I got confused, because end user will have wishlist but she was explaining connection between broker and exchange. But then Gaurav you point it and got that cleared at 22:10
Great Content
Why do we need cache for watchlist? Prices of the stocks which are in Watchlist will frequently be updated (every second) .
Is it correct or am I missing something
We can cache the stocks in the watchlist. Their prices will change but the stocks will change only once in a while.
Watchlists will be viewed frequently, so having the stocks cached makes sense.
I believe she means caching the list of stocks, not the underlying pricing data.
@@ITTIsiKhushi-keertipurswani
Oh, now I got it.
Ticker names could be cached and getting the name of the ticker( to fetch the prices from separate api) from cache will be faster compared to fetching the tickers name from DB.
Understood.
Thank You 🤗
Damn. I replied from the wrong account by mistake 😅
@@KeertiPurswani what exactly does once in a while mean? Lets say some traders update their watchlist every day, will it still be feasible to maintain watchlist in cache?
Video could have been bit more edited in order to reduce the length. Too much of internal clarification is going on which could be edited.
Nithin kamath entered the comments section.
It'll be an honor haha 😁
gaurav can you do a mobile banking and credit card application please high level?
what is gkcs full form?
here is the service that is being provided by seperate server
👍
every 50ms would be terrible. high frequency trading is in nano seconds :)
Zerodha does not provide HFT to end users (nano seconds is impossible on the poor connection speeds we all have, including 5G).
Why not bring engineers of zerodha to do such a video? Bringing random people with no idea of actual broker systems does give much knowledge.
watch dogs 😂