For all the "talks" and whatever people do trying to get the big idea out, theres something nice about being able to just watch a guy use the tool and do it. This was extremely helpful
I really appreciate the workflow of writing the access patterns out and writing the model step by step. It’s really difficult to find design patterns for typical entity relationships, and this is extremely helpful. I’m still trying to figure out when a GSI/LSI is warranted, it seems like they make many to many relationships easy. It seems like Table/PK/SK are the only real hierarchies to index something, and if you needed to access it from the other direction you either need another index, duplicate/pointer nodes and multiple requests, or a scan. It seems dynamodb is really a cache for direct lookups and there are costs to maintain that cache for instant lookups. My current table model is to use an ID appended to sort keys so that I can use Queries, just haven’t seen this model anywhere and I’m trying to figure out if I’ll run into issues - EDIT it looks like Alex DeBrie’s single table design video shows this pattern around 53:00, exactly what I was thinking
For a dolt like myself this video really helped me reason and make peace with my 'normalization' biased thinking. I still hate myself a little (for going to the dark side) but I am starting to see the advantages in single table design. Thank you for your efforts.
I'm curious about the ids for customer, product, warehouse, etc. How do we ensure there aren't id collisions across entities? Are these ids UUIDs and therefore all entities can take their ids from the same "stream" of generated ids, and we don't worry about separate sequences of ids for each entity?
My team has been delaying this because of the complexity. Now I know how to convince them step by step. Crack the entity relation ship and then the access pattern and then start modeling. Now see if the model satisfies the access pattern. Boom !! Thanks for this video.
I have seen almost all videos of ric and alex but there is a common use case related to get all data of a particular entity which is similar to select * from table. how should we approach that? if we keep it in a partition it can create a hot partition issue
00:05 Focus on single table design benefits 01:52 Challenges with multi-table data model and large hierarchical documents. 05:12 Understanding use case and data modeling for single table 06:48 Entities include customers, products, orders, invoices, payments, warehouses, and shipments 10:05 Decorating objects with attributes and indexing for efficient querying 11:43 Designing the data model using NoSQL Workbench 15:13 Implementing access patterns for querying DynamoDB tables. 16:57 Iterating through result set and switching on object type for dynamic wrapping. 20:09 Query table by product id and warehouse id 21:43 Redesigning the data model to handle complex access patterns. 24:47 Consider adding individual items to separate partitions for efficient indexing. 26:28 Payments are added to orders and entered on the invoice item 29:53 Using partitions in DynamoDB Single Table Design 31:25 Consider consolidating related data into a single object when it makes sense. 34:36 Creating a new index and adding index attributes to the data model 36:20 Structuring data for different customer needs 39:36 Querying GSI1 for different access patterns 41:13 Iterating through shipments and updating status using secondary index 44:26 Implementing Time To Live (TTL) for automatic data removal after a specified time period 45:52 Different access patterns may require varied amounts of metadata, so it's important to design data models that minimize unnecessary data retrieval. 49:18 Using selective prefixes for querying customer data in Amazon DynamoDB 50:48 Understanding data reading and filtering in DynamoDB 54:10 Efficient data modeling is essential for scaling systems and reducing inefficiency. 55:46 Single table design model represents the design implemented thousands of times at Amazon and across the customer base. Crafted by Merlin AI.
if i have to fetch details of a customer c#123 (like name, email etc) and all their orders, is it possible to fetch that in a single query? I can do that in two calls by first fetching customer data via PK=c#123 AND SK=c#123, and later via PK=o#123 AND SK=o#123
I realize this is an old video now, but it would be very helpful if you provided a link to the final data model file you import. You move through the data pretty fast and being able to check it out at my leisure would be very helpful. It's still a great video though!
I think you can add an attribute called "shop" for example on each customer and product record and create a gsi where pk = shop, sk starts with c# or p#
Great video ! I have a question though, I'm new to DynamoDB so there is one missing piece I did not get, in 20:15 if I have a product Id I can query for that product all warehouses "begins with w" but how do I get both quantity available and warehouses' name in a single query (without having replication of attribute)? Do I need to query warehouses' name for each Id ?
Create custom index or use scan with primary id and then sort the data. Second one is expensive and first one is custom pre build with the usecase in mind.
That helps to query data by not using primary keys and for custom purpose like ' give me the top warehouses running out of product data' etc. We can't build different tables for our custom usecases so effectively model data in few tables and build index to support custom query or use scan.
What if for example the customer for some reason edits his name how will i update the customer name in all other records for example in orders where it has the old customer name?
@4§4§r1 But how will this approach store the customer name in the order record? The customer might have multiple orders stored in a list or a map in the customer record and i can't make a gsi on a list or map to be able to project his name. I don't think there is a way around this except for fetching all customer orders and updating each order to change the customer name.
@Veerabramhendra Swamy Pothuluri I actually ended up considering this solution, i guess after all i can't optimize for both reads and writes. I will either store customer name in each order and optimze for reads but then i'll have to deal with updating each record if the customer edits his name, Or just store the customer id and for every order record i'll have to fetch the customer name which is also a bit too much. But anyways thanks for your response i really apperciate it.
@@ahmadnabil2441 also worth noting that in some cases you want to keep the previous customer name so copying it to record can actually be feature especially in case of invoices (eg. when customer is company and you need to keep the previous name intact on old records)
![AWS re:Invent 2019: [REPEAT 1] Amazon DynamoDB deep dive: Advanced design patterns (DAT403-R1)](/img/n.gif)
For all the "talks" and whatever people do trying to get the big idea out, theres something nice about being able to just watch a guy use the tool and do it. This was extremely helpful
100% agree!
So far the best video to understand single table design on dynamoDB, learnt a lot about this powerful db
You have got an enviable workplace Rick. One of the best sessions for DynamoDb single table modelling.
I assumed it was a background. The light on the face has cool colours, whereas the light in the background has warm colours.
I really appreciate the workflow of writing the access patterns out and writing the model step by step. It’s really difficult to find design patterns for typical entity relationships, and this is extremely helpful. I’m still trying to figure out when a GSI/LSI is warranted, it seems like they make many to many relationships easy. It seems like Table/PK/SK are the only real hierarchies to index something, and if you needed to access it from the other direction you either need another index, duplicate/pointer nodes and multiple requests, or a scan. It seems dynamodb is really a cache for direct lookups and there are costs to maintain that cache for instant lookups.
My current table model is to use an ID appended to sort keys so that I can use Queries, just haven’t seen this model anywhere and I’m trying to figure out if I’ll run into issues - EDIT it looks like Alex DeBrie’s single table design video shows this pattern around 53:00, exactly what I was thinking
For a dolt like myself this video really helped me reason and make peace with my 'normalization' biased thinking. I still hate myself a little (for going to the dark side) but I am starting to see the advantages in single table design. Thank you for your efforts.
I am japanese.I was impressed with this video.
Excellent video for DynamoDb single table modeling. The presentation style was very interesting, loved it, thanks!
I'm curious about the ids for customer, product, warehouse, etc. How do we ensure there aren't id collisions across entities? Are these ids UUIDs and therefore all entities can take their ids from the same "stream" of generated ids, and we don't worry about separate sequences of ids for each entity?
My team has been delaying this because of the complexity. Now I know how to convince them step by step. Crack the entity relation ship and then the access pattern and then start modeling. Now see if the model satisfies the access pattern. Boom !! Thanks for this video.
It was understandable in the first half, but got unclear real quick... thanks for the efforts still.
You are a GOD!!!!! Hands down, this is the best explanation out there!
Thanks for your video, it's very helpful!! this topic is pretty complicated especially if you are from sql like me
Would be great if you would share data model for Nosql workbench.
The data model used is this:
github.com/aws-samples/amazon-dynamodb-design-patterns/tree/master/examples/an-online-shop
would you please give the workbench data model files
I have seen almost all videos of ric and alex but there is a common use case related to get all data of a particular entity which is similar to select * from table. how should we approach that? if we keep it in a partition it can create a hot partition issue
00:05 Focus on single table design benefits
01:52 Challenges with multi-table data model and large hierarchical documents.
05:12 Understanding use case and data modeling for single table
06:48 Entities include customers, products, orders, invoices, payments, warehouses, and shipments
10:05 Decorating objects with attributes and indexing for efficient querying
11:43 Designing the data model using NoSQL Workbench
15:13 Implementing access patterns for querying DynamoDB tables.
16:57 Iterating through result set and switching on object type for dynamic wrapping.
20:09 Query table by product id and warehouse id
21:43 Redesigning the data model to handle complex access patterns.
24:47 Consider adding individual items to separate partitions for efficient indexing.
26:28 Payments are added to orders and entered on the invoice item
29:53 Using partitions in DynamoDB Single Table Design
31:25 Consider consolidating related data into a single object when it makes sense.
34:36 Creating a new index and adding index attributes to the data model
36:20 Structuring data for different customer needs
39:36 Querying GSI1 for different access patterns
41:13 Iterating through shipments and updating status using secondary index
44:26 Implementing Time To Live (TTL) for automatic data removal after a specified time period
45:52 Different access patterns may require varied amounts of metadata, so it's important to design data models that minimize unnecessary data retrieval.
49:18 Using selective prefixes for querying customer data in Amazon DynamoDB
50:48 Understanding data reading and filtering in DynamoDB
54:10 Efficient data modeling is essential for scaling systems and reducing inefficiency.
55:46 Single table design model represents the design implemented thousands of times at Amazon and across the customer base.
Crafted by Merlin AI.
It took me at least two watches of this video to wrap my head around it
Thank you Rick! Awesome stuff!
Excellent ! 👏🏻 is it possible that you can share the files of this example in workbench? To see in local environment. Pls 🙌🏻🙏🏻
This would be so wonderful! Please!
Yes please it would be great
They are available here: github.com/aws-samples/amazon-dynamodb-design-patterns/tree/master/examples/an-online-shop
@@saurabhshri thanks a lot !
if i have to fetch details of a customer c#123 (like name, email etc) and all their orders, is it possible to fetch that in a single query?
I can do that in two calls by first fetching customer data via PK=c#123 AND SK=c#123, and later via PK=o#123 AND SK=o#123
I realize this is an old video now, but it would be very helpful if you provided a link to the final data model file you import. You move through the data pretty fast and being able to check it out at my leisure would be very helpful.
It's still a great video though!
At 10:26, does the sort key really allow querying with an "in" condition? AFAIK, we cannot currently do this.
Awesome Rick
I feel like a kid who has just discovered a new amazing toy
I heard a saying that DDB transactions feature eliminates the need to design single table. Is that true?
Hey, very interesting approach, what would be the best method for a get all Customer or get all Product for example?
I think you can add an attribute called "shop" for example on each customer and product record and create a gsi where pk = shop, sk starts with c# or p#
Great video ! I have a question though, I'm new to DynamoDB so there is one missing piece I did not get, in 20:15 if I have a product Id I can query for that product all warehouses "begins with w" but how do I get both quantity available and warehouses' name in a single query (without having replication of attribute)? Do I need to query warehouses' name for each Id ?
Create custom index or use scan with primary id and then sort the data. Second one is expensive and first one is custom pre build with the usecase in mind.
Awesome lesson!
SUPER VIDEO!!!!!!!!!!!!!!!!!!!!!
Is the sound ok over there? Here it is kinda breaking up
For your GSIs then, do you always use an ALL projection? It seems like to use patterns like this you can't limit the projection
You could still create GSIs for quick actions that are done often like getting user account info
Is there any chance to get the DB modeling file?
What tool is he using that says AWS visualizer?
NoSQL Workbench - docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.html
I dont understand why Rick creates a second GSI. Can someone please explain it to me? Thanks
That helps to query data by not using primary keys and for custom purpose like ' give me the top warehouses running out of product data' etc.
We can't build different tables for our custom usecases so effectively model data in few tables and build index to support custom query or use scan.
Is it possible to get an import file of this schema?
What if for example the customer for some reason edits his name how will i update the customer name in all other records for example in orders where it has the old customer name?
@4§4§r1 But how will this approach store the customer name in the order record?
The customer might have multiple orders stored in a list or a map in the customer record and i can't make a gsi on a list or map to be able to project his name.
I don't think there is a way around this except for fetching all customer orders and updating each order to change the customer name.
@Veerabramhendra Swamy Pothuluri I actually ended up considering this solution, i guess after all i can't optimize for both reads and writes. I will either store customer name in each order and optimze for reads but then i'll have to deal with updating each record if the customer edits his name, Or just store the customer id and for every order record i'll have to fetch the customer name which is also a bit too much. But anyways thanks for your response i really apperciate it.
@@ahmadnabil2441 also worth noting that in some cases you want to keep the previous customer name so copying it to record can actually be feature especially in case of invoices (eg. when customer is company and you need to keep the previous name intact on old records)
@@GrizzlyNetch Oh, I never thought of that my usecase is actually kinda similar. Thank you very much for the advice.
Hi Rick, maybe you could post the data model file so that we can play around on our own? :)