This is literally what I needed today. Cramming this playlist hopefully offer pulls up, I never comment on posts but u are the goat bro the goat broski if I get this offer I will send u some only fans money
Hey Jordan! Thanks as always for these awesome videos! I was wondering if you could do a quick video about how to actually structure and talk about these on an interview. Is there a structure to it that you do (or have seen), such as laying out requirements first, then doing some considerations, then diving into the design? Is it really not one size fits all? Either way could be interesting to see what tips you might have around this. Keep it up, you're incredibly helpful!
The other video I’d like to see: A distributed system for generating unique IDs, akin to Twitter Snowflake. Also with the functional requirement of how to allow people to bring along their own IDs.
Thanks for the video Jordan. How is polling going to work? Is there a cron job or a scheduler running every x minutes in the flink? Moreover maintaining the doubly linkedlist in the order of create time stamp would be log(n) right, as each event reaching flink could be out of order, right?
Yeah something like that, or you could just say something like on a new event, if we haven't polled in x amount of time, poll again. I would think that events reaching Flink should be coming in order on a timestamp per partition, so we could always just do a flink node per partition to maintain that invariant.
The assertion behind the need for a derived Pending Transactions cache - that reads will hurt write throughput due to row locking - is not necessarily true if you're using a DB with MVCC (like Spanner or Cockroach). So I question the need for the separate cache.
The other reason is that you then need to run a query on disk to figure out all of the pending transactions. I'd rather just have them all precached, but agreed that if you're using snapshot isolation locking is unnecessary for such a read
@@jordanhasnolife5163 could add a local covered index to speed up the query and ensure consistency, which would slow down writes a bit but per the original requirements that's not a problem. In any case pending transactions would be a great topic to deep dive in a real interview and discuss tradeoffs, so thanks for calling it out explicitly in your video
Can you make a video on designing Spotify? Func Requirements: - Ability to play any song. super low latency while playing any song. - User can create playlist, share playlist. - Follow other playlist, artist, get notified for any song updates by artist or in a playlist.
At least off the cuff I'll say 1) CDNs and precaching when going down a playlist 2) Use a database 3) This feels like twitter You think there are any other unique pieces to it?
@@jordanhasnolife5163 how about live streaming & podcasts? I believe that this is similar to Zoom. But would like to know if there any optimizations which can be done?
Thanks for the great video! Question about web hooks. So the Payment system is listening for web hook callbacks and the polling mechanism is only triggered when a pending payment hasn’t received a callback in a specified amount of time? Is that the idea?
I have a dumb question. Why would row read locks on the pending payments slow down the write throughput of the table, given the writes' idempotency key are different from the pending ones?
Great point, there probably wouldn't be many conflicts IRL, but the reads themselves would be quite expensive and would take resources away from the DB
thanks Jordan I have been watching your sd video each week have two qq regarding your design 1. could we use snowflake algorithm to generate id as idompotence key ? 2. was Flink processing part of payment service code ? if so, for fault tolerance case where payment service was down how is it going to affect flink processing ? thanks
1) Not familiar with this method, feel free to send me a link to what it is 2) Not sure what you mean by this question. Flink is just getting data from our payment db, and occasionally polling stripe to see the status of it, it is independent of any synchronous operation to do with the payment service.
@@jordanhasnolife5163 thanks for replying 1) en.wikipedia.org/wiki/Snowflake_ID 2) let me rephrase my question a little bit where was the application code to generate idempotence key logic and save to payment as one microservice and flink process another microservice or they are all clustered as one service as in payment service. if both processes are treated as one service application code and payment service was down, flink process will also halt right
I don't really know how much there is to elaborate on here beyond what I've discussed in my concepts videos. I'd take a look at something like debezium.
17:18 - should pending payments that are `not recognized` by stripe at poll time really just be deleted from the payments table? this case might require special processing since at this point, the payment has a local DB status of `pending` but stripe has not recognized this payment. What would be a possible solution for this inconsistency?
I don't really think there is any solution, network requests to stripe can always fail. Do we want to delete the event? Maybe not but then we may find ourselves doing a lot of polling after a while.
I think Cassandra's "strong consistency" is probably quorum consistency. I'd look into spanner, cockroach, yugabyte, as it seems they lean towards using distributed consensus within a replication group.
Oh man I'll have to look into this one, you may be aging yourself by asking for an RSS feed and I may be aging myself by saying I've never used one lol
Thanks for the video! Why not just use zookeeper to give us a monotonically increasing u64 for the idempotency key? This way we are guaranteed to not have any conflict, also u64 should be enough till the end of times.
1) using a monotonically increasing sequence number implies that all writes must go through the same choke point (meaning you can't shard zookeeper, which is potentially fine if we really don't care about performance) 2) We basically do this anyways, as our payments db is basically using a consensus algorithm, making it effectively the same as zookeeper
Thanks Jordan for your awesome video! Hope I can see the topic about 'design some meeting scheduler' thing one day~~
This is literally what I needed today. Cramming this playlist hopefully offer pulls up, I never comment on posts but u are the goat bro the goat broski if I get this offer I will send u some only fans money
Haha please take the only fans money and donate it to charity
One more banger system design video!
the cache design that you mention with the doubly linked list and hashmap is basically the implementation a LRU cache
You're correct
Hey Jordan! Thanks as always for these awesome videos! I was wondering if you could do a quick video about how to actually structure and talk about these on an interview. Is there a structure to it that you do (or have seen), such as laying out requirements first, then doing some considerations, then diving into the design? Is it really not one size fits all? Either way could be interesting to see what tips you might have around this. Keep it up, you're incredibly helpful!
Here ya go m8
ua-cam.com/video/IY2EPjShgc4/v-deo.htmlsi=Xw4uwvd4iDBbpp_w
You can always just ask your Interviewer too. Hey is it ok if I start with x?
@@jordanhasnolife5163 oh shit I missed that, thanks! 🙏🏼
The other video I’d like to see: A distributed system for generating unique IDs, akin to Twitter Snowflake. Also with the functional requirement of how to allow people to bring along their own IDs.
This does feel somewhat similar to what we do in the payment gateway video, shard the key range, allow users to bring their own
First! Happy Saturday!
Please make a video on design aws cloud watch
See distributed logging and metrics video
why are you so smart, my love Jason?
😙
Thanks for the video Jordan. How is polling going to work? Is there a cron job or a scheduler running every x minutes in the flink? Moreover maintaining the doubly linkedlist in the order of create time stamp would be log(n) right, as each event reaching flink could be out of order, right?
Yeah something like that, or you could just say something like on a new event, if we haven't polled in x amount of time, poll again.
I would think that events reaching Flink should be coming in order on a timestamp per partition, so we could always just do a flink node per partition to maintain that invariant.
The assertion behind the need for a derived Pending Transactions cache - that reads will hurt write throughput due to row locking - is not necessarily true if you're using a DB with MVCC (like Spanner or Cockroach). So I question the need for the separate cache.
The other reason is that you then need to run a query on disk to figure out all of the pending transactions. I'd rather just have them all precached, but agreed that if you're using snapshot isolation locking is unnecessary for such a read
@@jordanhasnolife5163 could add a local covered index to speed up the query and ensure consistency, which would slow down writes a bit but per the original requirements that's not a problem. In any case pending transactions would be a great topic to deep dive in a real interview and discuss tradeoffs, so thanks for calling it out explicitly in your video
Can you make a video on designing Spotify?
Func Requirements:
- Ability to play any song. super low latency while playing any song.
- User can create playlist, share playlist.
- Follow other playlist, artist, get notified for any song updates by artist or in a playlist.
At least off the cuff I'll say
1) CDNs and precaching when going down a playlist
2) Use a database
3) This feels like twitter
You think there are any other unique pieces to it?
@@jordanhasnolife5163 how about live streaming & podcasts? I believe that this is similar to Zoom. But would like to know if there any optimizations which can be done?
Thanks a lot for another amazing video... I've a question, how does the payment reaches seller?
Well I guess thats a detail for tipalti, but Amazon probably makes batch payments every month to them via an ACH (wire) trabsfer
Thanks for the great video! Question about web hooks. So the Payment system is listening for web hook callbacks and the polling mechanism is only triggered when a pending payment hasn’t received a callback in a specified amount of time? Is that the idea?
Yep!
I have a dumb question. Why would row read locks on the pending payments slow down the write throughput of the table, given the writes' idempotency key are different from the pending ones?
Great point, there probably wouldn't be many conflicts IRL, but the reads themselves would be quite expensive and would take resources away from the DB
thanks Jordan I have been watching your sd video each week have two qq regarding your design
1. could we use snowflake algorithm to generate id as idompotence key ?
2. was Flink processing part of payment service code ? if so, for fault tolerance case where payment service was down how is it going to affect flink processing ?
thanks
1) Not familiar with this method, feel free to send me a link to what it is
2) Not sure what you mean by this question. Flink is just getting data from our payment db, and occasionally polling stripe to see the status of it, it is independent of any synchronous operation to do with the payment service.
@@jordanhasnolife5163 thanks for replying
1) en.wikipedia.org/wiki/Snowflake_ID
2) let me rephrase my question a little bit where was the application code to generate idempotence key logic and save to payment as one microservice and flink process another microservice or they are all clustered as one service as in payment service. if both processes are treated as one service application code and payment service was down, flink process will also halt right
can you make a separate video elaborating Change Data Capture part ? like log based , trigger based ...
I don't really know how much there is to elaborate on here beyond what I've discussed in my concepts videos. I'd take a look at something like debezium.
17:18 - should pending payments that are `not recognized` by stripe at poll time really just be deleted from the payments table? this case might require special processing since at this point, the payment has a local DB status of `pending` but stripe has not recognized this payment. What would be a possible solution for this inconsistency?
I don't really think there is any solution, network requests to stripe can always fail. Do we want to delete the event? Maybe not but then we may find ourselves doing a lot of polling after a while.
What happens if Flink cache fails? I think we will somehow have to redrive the CDC stream to repopulate the new cache instance?
Please see the flink concepts video. State is periodically checkpoibted to s3
Could you suggest a database that would match the consistency requirements? Or are we rolling our own?
I see Cassandra can be configured into a strong consistency mode?
I think Cassandra's "strong consistency" is probably quorum consistency. I'd look into spanner, cockroach, yugabyte, as it seems they lean towards using distributed consensus within a replication group.
can you do privacy/visibility controls system design?
Perhaps, how do you see this one being a challenge after we put everything in a strongly consistent table?
In a future video, could you do an RSS newsfeed aggregator? Maybe throw keyword search in there.
Oh man I'll have to look into this one, you may be aging yourself by asking for an RSS feed and I may be aging myself by saying I've never used one lol
Thanks for the video! Why not just use zookeeper to give us a monotonically increasing u64 for the idempotency key? This way we are guaranteed to not have any conflict, also u64 should be enough till the end of times.
hey friend, can you please explain what is u64? is it like a uuid?
@@lalasmith2137 haha sorry, an unsigned 64 bits integer
@@huguesbouvier3821 thank you for clarifying that, helped me understand your answer :)
1) using a monotonically increasing sequence number implies that all writes must go through the same choke point (meaning you can't shard zookeeper, which is potentially fine if we really don't care about performance)
2) We basically do this anyways, as our payments db is basically using a consensus algorithm, making it effectively the same as zookeeper