AWS DynamoDB Schema Design | How to choose the right key
Вставка
- Опубліковано 1 лип 2024
- AWS DynamoDB Tutorial. DynamoDB is a powerful fully managed NoSQL database. Part of the complexity in using the service is related to choosing a partition key that not only serves your access patterns, but prevents you from having scaling related headaches in the future. In this video, I talk about the factors you should consider when choosing a partition key. I start out by briefly talking about why choosing the right key is important, followed by defining partition and sort keys. Afterwards we go into key partitioning behavior and a feature called adaptive capacity. Finally, I conclude with some general guidance on when you should choose one key structure over another.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
00:25 Why picking the right key is important
01:10 Primary Key, Partition Key, Sort Key
09:24 When to use what key
🎉SUPPORT BE A BETTER DEV🎉
Become a Patron: / beabetterdev
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
📚 References:
Getting started with AWS: • Introduction to AWS | ...
DynamoDB Global Secondary Indexes - • What is a DynamoDB GSI...
DynamoDB Local Secondary Indexes - • What is a DynamoDB LSI...
DynamoDB Accelerator (DAX) - • Improve DynamoDB Perfo...
SQL vs NoSQL explained - • SQL vs NoSQL Explained
Choosing the right key - aws.amazon.com/blogs/database...
Understanding Adaptive Capacity - docs.aws.amazon.com/amazondyn...
☁Topics covered include:
- Primary Key
- Partition Key
- Sort Key
- Range Key
- Partitions,
- Partition Splitting
- Read Capacity Units (RCU)
- Write Capacity Units (WCU)
- Throttling
- Adaptive Capacity
- Burst Capacity
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#DynamoDB
#PartitionKey
#SortKey
This is one of the most useful video I found on dynamodb keys. You made me your subscriber
Thank you for the kind words. Its comments like these that motivate me to keep making more videos. Thanks again and take care.
This is by far the best video I found for DynamoDB Keys!
Am a veteran RDBMS guy and new to NoSQL. I got to say, the way your describe NoSQL stuff is just phenomenal.
Thank you so much for the kind words!
Best Dynamo DB Schema Tutorials with precision knowledge providing all the use cases. Many Thanks.
Thank you so much for your kind words!
You're such a gifted teacher. It's not easy to grasp all this information from the documentation. Thanks for the amazing video.
Thank you so much for your kind words. I actually kind of enjoy reading documentation and I'm glad I can translate that into easy to understand examples for all of my viewers.
Thanks again!
This was excellent - wish I had watched it 6 months ago! I was actually searching for "cardinality" and your vid came up among the first results.... Thanx again.👍
Great job explaining the topic and exploring the limitations. The way you explain a complicated subject with simplicity without sacrificing knowledge is sheer elegance. Keep it up.
Thanks so much Lee! Your comments means a lot to me :)
Yes BABDev: simplitivity without sacrificing knowledge...spot on! I like the way you explain everything. When I'm learning from a tutorial I HATE having to spend half the time googling what was mentionef but not explained....thanks for NOTsoing this...it really helps.
Ps...great voice too...clear&well modulated. Just easy to listen to.
Man you really simplified it. The original documentation will take you in circles all over the place. Fantastic video.
Thank you so much! I'm glad you enjoyed the video and helped you understand the concepts.
Daniel
The whole hot partition concern and the WCU / RCU hard limits are pretty much a thing of the past. DynamoDB will automatically reshard your partitions behind the scenes down to a single item if the disproportional velocity access is there. It means you'll only reach hard limit throttling if you have a single item being responsible for more than 1000 WCU / 3000 RCU.
You are awesome and one of the best teachers I have ever seen. You boil down the concepts into simpler concepts extremely well. API Gateway tutorials please (I can even pay you for that :P)
Thanks Suresh for such kind words. It is comments like yours that motivate to keep making content. Thank you!
Thanks for explaining in a simplified way. Very useful
You're very welcome sumaiya!
Best DynamoDb explained ever.
You just won a subscriber. High-quality stuff (y)
Thanks Akash! Glad you enjoyed.
Very clear, easy to understand, thank you!
Glad it was helpful!
Fantastic video! Easy to learn. Thank you 🙏
Great stuff! Very helpful, thanks.
Excellent video. Good speaker and provides content very well! Easy to learn from!!
Much appreciated!
Simple and to the point explanation...Keep up the good work
Thanks a lot!
I agree with all the comments here. Nice work describing this in simple terms.
Thank you!
Very well explained. Thank you!
You're very welcome Zain!
Splitting up the RCU to depict the partition capacity is by far the best way I seen hot partition be explained.
Thank you!
@@BeABetterDev Thanks to you I got my Dev Associate!!!
Awesome tutorial !!
Thank you! Cheers!
4 years alter... this run through is still very helpful. 👍
you deserve more subscribers .. keep up good work
I appreciate that!
very well explained.. all stuff no fluff.
Thanks ashutosh!
Great video. Thank you so much!
Thank Hamza, super glad it was helpful!
Impressed.... Very informative
Glad you liked it!
These are really good mane, keep it up 👍🏻
Thanks Michael!
Excellent video . It really helps. Thanks a lot.
Glad it helped!
How do you not have more subs? your channel is great!
Keep it up mate!
Thanks a ton!
Very well explained!
Thanks vimal!
Awesome!! Thanks for this. It was very helpfull.
You're welcome!
Very nicely explained.
Thank you! 🙂
Thank you for this tutorial 🙏🙏🙏
You're so welcome Jeffry!
Very nicely explained. Keep it up :)
Thanks Akshay!
Awesome video, keep the good work!
Thanks Gustavo! Really appreciate the kind words.
Thank you for the info!
This was super useful, would be awesome if you could do a video on GSI and how that works, can't seem to wrap my head around it fully
Thanks Alicia. You''re feedback means a lot. I have one coming out this weekend on DynamoDB Autoscaling, but definitely plan on doing one on GSIs in the next month or so. Thanks again!
Hey alicia, check out this new video I just posted on GSIs! ua-cam.com/video/ihMOlb8EZKE/v-deo.html
Excellent video on DynamoDb partition key concepts , it would really help. If you could provide any video on DAX whenit can be used and its pros and cons
Thank you for the kind words. A video on DAX is coming soon!
Awesome .. Simply put
Thank you!
great explanation!
Thanks Ishaan!
Nicely explained! Subscribed :)
Awesome, thank you!
Very helpful thank you!
You're very welcome!
Very useful.. thanks for the video, AWS docs are jargon :P
Glad it helped!
Hey! Your videos are very helpful and I learn a lot of things!
For me and maybe another people it might be interesting to see an end to end project. You don't have to implement it but something like a general concept would be awesome.
For example something like: Deploy a website and a REST service via AWS. Which services should we use and how to secure the backend and how to connect the REST API to a DB. Things like this would be absolutely awesome :)
I'll definitely follow you and watch the other tutorials.
Cheers mate :)
Thanks Neal! An End to End video has been on my radar for a while. Thanks for the suggestion!
@@BeABetterDev An additional suggestion that might be easier and also makes a lot in for example the whole aws rds context is a tutorial about bastions/jump boxes. I. e. host an RDS Aurora db in a private vpc and then you create a jumbox to connect to this db via SSH.
Have a great day :)
@@XxGabberlordxX3:24
Thanks, it's a great video :)
Glad you liked it!
Great Video, thank you for all the information! I have a question about filtering - which solution would you approach if you wanted to filter with multiple conditions? Equivalent to: WHERE country=„…“ & age=„…“ & … and so on (note: country and age are not pk or sk and without using filterExpressions)
Or is there not a nice solution with dynamodb to that and aurora would be a better choice?
Thank you in advance
Great video Thanks
Thanks Ray!
Thanks for wonderful video
You're very welcome kunal!
This is super useful, please create a video on local secondary index as well
Thanks Rama! I'll add LSIs to my backlog of ideas. Cheers.
Hi Rama, in response to your feedback, I just created a video on Local Secondary Indexes - check it out here: ua-cam.com/video/Y8gMoZOMYyg/v-deo.html
Very Nice Explanation :)
Thank you Sagar!
Thank you. Well explained. Another subscriber here :-)
Thank you so much Venkat! Welcome to the channel.
Most useful video
Thanks Yogitha!
Thanks amazing video
Thank you !
Great content
Thanks Jeffin!
Thank u for sharing. Wud b cool if u mention filtering in the video
Great suggestion!
Great video, I got a question through an example If I had a sign up method in app and I can do it by email or phone number I was thinking about primary key because of cognito Id user, and that user sell products using many categories can have those products and categories under same table and row and query it by it's I'd, it's a good practice ? thanks
So does querying with parition and sort key will take logn time? Partition key will determine with hashing function the correct partition and then in that partition we will use sort key(log n time, where n is rows with same partition key value) to get to the reuqired row?
wow only 1.93K subs.. what a crime. Subbed!
Thank you so much Calvin for your kind words!
Very nicely explained indeed. I have seen some examples where the Partition Key is an UUID value. However, in some cases, I need my primary key to be an auto-increment integer like the one many SQL databases provide. What is your suggestion?
Hi Kyriacos,
Unfortunately auto-increment isnt supported in DynamoDB. Generally speaking, in a NoSQL database it is advised to use a meaningful primary key so you can quickly retrieve your data by said key. Your access pattern should define what your primary key should be (even a composite key could be used).
Your requirements make me think a traditional relational database could be better suited for your use case.
Hi. When designing a DB schema in DynamoDB how do you store the traditional lookup values like countries? In it's own table country with countryCode, countryName attributes? What's the best practice here? Thanks.
Excellent video and information, keep doing such quality content
Thanks you so much!
So in an amplify project for example, this schema is defined in the graphql schema yes? Do you have any such videos. I found a very good post on Dev.to from Nader Dabit about creating a robust graphql schema with directives, thanks!
Really good
Thanks!
Great video. Just one doubt. How to design my table if i have more than 2 fields for uniqueness (let's say 3 fields defining uniqueness. For example, in vehicle, make+model+year is unique)
nice video, thanks u!!!
Glad you liked it!
thanks dude
Thank you
You are very welcome!
Excellent video !! Do you have any video on data modelling a relational database in DynamoDB
Hi Ajay, not yet but thanks for the great tip!
In my case i want to scan all data each time to compare with my response to know where update occur or insert and according to that i want to perform other requests...so what could be the best possible way?
Nice one !! i just subscribed
Thank you!
Is it possible for you to create a video on how to create DynamoDB table - having LSI and GSIs through Cloudformation script and add items in it using put_item from boto3 python SDK?
you rock!
Suppose if we design sort key as countrycode#date, how we will be able to sort date only as we have country code along with it
how to use update method with combination of partition key and sort key?
Thank you, just subbed. I'm about to get my hands dirty with AWS for the first time, use case is building an API and a PWA with React, was thinking of using the serverless framework, some advise to steer me in right direction would be greatly appreciated, I'm also doing my own research so it looks like I will need the combination of the following..API Gateway, Lamda functions, Dynamo DB, Cognito for backend, S3 for hosting, serverless for my framework, I'm sure theres more, thank you!
Hi Lardosian! Welcome to the channel.
Great to hear you're getting familiar with AWS.
In terms of your use cases, It sounds like you've identified the right AWS technologies for your use cases. Here are some additional videos I think could be helpful.
IAM - IAM permissions are the bread and butter of AWS. You should get very familiar with them - ua-cam.com/video/y8cbKJAo3B4/v-deo.html
Lambda Introduction - ua-cam.com/video/iUIWG0h2D84/v-deo.html
Building a REST API with API Gateway + Lambda - ua-cam.com/video/uFsaiEhr1zs/v-deo.html
Basic DynamoDB Operations - ua-cam.com/play/PL9nWRykSBSFgmcJMpr7XAk3BC0VPAK_i-.html
Hope this helps!
@@BeABetterDev Thank you very much, apologies for the late reply.
Sorry, I would like to ask a question. At 6:18, you said that item with the partition key 3 will be stored in the same partition as item with partition key 1. But aren't they having different hash value after you put their partition key into the hash function? So why did they end up in the same partition?
Hi 黃翌軒,
Depending on the size of your table, DynamoDB may store values with a different partition key on the same physical partition. As your data size or throughput demands grow, DynamoDB can re-split these partitions into smaller ones.
Hope this helps.
Is there a reason to use DAX over CloudFront??
Thanks for this very insightful session. Just would like to know where I can ask questions. I have one doubt about PutItem, which is not related to this video.
Thanks Abhay! feel free to ask here
I have asked it in your other video related to inserting items in DynamoDB.. Thanks
also, with partiQL you can query table with a value without cretaing a GSI on that column. is that a new feature?
Hi Rohit, the thing you need to be careful about using PartiQL is that performing queries are non-indexed columns can result in a scan operation instead of a query. This can cause high costs and poor performance for these types of operations. Hope this helps clarify.
Please can we get a tutorial. Where I can use curl to create an presigned url for s3 bucket object to load an file and download a file .
6:20 How does the item with Account Id = 3 goes into Partition 1? Shouldn't it go to Partition 3 since the Account Id attribute is defined as partition key?
IN my understanding, When the hash function is applied on the account Id it generated the target as partition 1. E.g. id % 2 ==0 will go in partition 2 and rest in 1.
Hope it helps
Hey! how to find my table's details or description on AWS DynamoDB?
Hi Mohammed, this should be in the Overview tab on the console after clicking the table name.
Very Knowledgeable video Clare my table schema confusion. Please do share the Github link for code reference for serverless application if you have any..
Thank you Rhushikesh! I will post github links soon. :)
How did you define a date as sort key? I do not see type date. Do you define date as string?
Hi Rohit date is just a string in this
Is GSI a new feature since it’s not mentioned in this informative video
Yep that sounds correct
Hey there! I have a question. What if I set RCU to 1? Does this mean that in this case all data will be stored on the one partition? Also, does this mean that the number of partitions for my table will be calculated from the RCU that I choose?
Hi Io,
Setting your RCU to 1 will severely limit your table's performance and result in throttling errors. Do note that a single partition has a maximum size of 10 gb. Your RCU selection does has some impact on the number of partitions, yes!
AWS Simplified Thank you! Yeah I realise that it’s not good to set the RCU to 1. I would just like to know if it’s possible to calculate the number of partitions by RCU or not
AWS Simplified also, I’ve tried to get throttling during the writing new records. I’ve got it however all records have been stored successfully. How can this be?
Good video.. after spending over a decade in SQL database.. its challenging to understand how SQL schema could be translated/redesigned to NOSQL. If you could make a video by taking example of AdventureWorks to show how you would design that use case in NOSQL, that would be great.
7:03 Why partion 3 row is in Partition 1 ? Shouldn't it be under a new partition 3 ?
Please help, this is my only doubt left .
Double this question
The location the partition lands on is independent on the data. Rows that have differnt partition keys can land on the same partition.
what if partition key itself is unique an there are 1 million records example email id as partition key in user table?will there be a problem with too many partitions?
Hi Gaurav,
Dynamo would split your data across multiple partitions in that case. There is no limit that I am aware of regarding number of partitions.
New subscriber here too. I'm really surprised you don't have a lot more views and likes. Your videos are very helpful and easy to follow. I don't know if this is a topic for another video or related to this video... does this key approach require one entity type per table ? I've been reviewing AWS Docs and tutorials (specifically this one - aws.amazon.com/getting-started/projects/design-a-database-for-a-mobile-app-with-dynamodb/4/) where multiple entity types can sit in one table. I would like to implement this approach but it doesn't translate well when you look at the console. Do you have a video that can help me out ?
Hi Saturday! Thank you for the kind words of support, it means a lot to me!
Regarding your question. It is possible to store multiple different types of entities within the same table. For example, in your example you have USER, PHOTO, REACTION, FRIENDSHIP entities. You can potentially store this data in one table by taking the following approach:
Set your primary key to be IDENTITYTYPE_HashKey. One of your columns could be a serialized object in JSON representation of the entity type. The one caveat here is that the serialized data in your JSON column would be unsearchable from the dynamo perspective (range key search, global secondary index search, etc). However, if you are looking to store flat data of different types in one table - this approach will certainly work.
Hope this helps and let me know if you have further questions!
Speaking of other strategies: How do you create composite primary key with 3+ columns?
Hi Miral,
I think it would be as simple as just appending each value using a delimiter such as value1-value2-value3. Keep in mind though this pattern requires you to know all these values before querying in order to find your records in Dynamo.
Hope this helps,
Daniel
@@BeABetterDev Thank for reply. That's the catch, the third column value is a timestamp and for that reason, can't perform BatchGetItem.
how to pagination using dynamoose
Most of the time, we depend of business rules.
I mean, recently, we create a table with just an uuid (for each object). Then later, The business would like to retrieve thoses objects but without knowing the uuid, they need to retrieve the object by the name And lastModificationDate (between two dates).
How do you manage that ? You have to create some gsi...
But we do not have always all the information when we start the archtecture. This is very complicated when your project evolves with anothers features that you do not anticipate.
This is my pain using dynamodb over mysql before.
The use case for Dynamodb is driven by high performance, so ultimately the business needs to understand the complexity of new queries. You can also utilise something like dynamo streams to update something like elastic search or ETL the data into a relational DB for complex queries.
How can a row with partition key "3" end up on Partition 1? 3:25
Hi Niclas. Partition assignment is completely random. Hope this clarifies.
how many items can I set in the same partition key?
Hi ruru,
There is no limit. However keep in mind that there is a 10gb max partition size in dynamo. I believe the partition splits afterwards. If you have too much throughput on the same partition key you make get throttling errors.
@@BeABetterDev thanks for answering me, the video was very helpful😁👍
Can you please share the slides
How many partition Key could dynamodb could have??
Hi Jivan, technically it is infinite.
I must be missing something fundamental about Dynamo because I keep looking up how to retrieve all items in my table in a sorted order, but I keep ending up on articles and videos like this where they talk about read capacity and partitioning. I want SELECT * FROM table ORDER BY time ASC. I get the feeling I could give everything the same partition key and unique sort key and have them come back ordered but that has to be the wrong way to achieve my goal.
I am not sure of your exact use case and why you need to do a Select * FROM table, but I am pure guessing from the way you state your comment that more than likely you are trying to apply a relational DB mindset onto a NoSQL database, in this case DynamoDB. If you want to expand your comment, I am happy to try and help you though.
@@NoSQLKnowHow I have been messing around with Dynamo to build a serverless version of my blog. The traffic I get would be well under the free tier limit for both Lambda and Dynamo, so I could have a blog hosted for cheap. As an example of what I'm trying to do, I would call Dynamo to retrieve my blog posts and have them return in order of the sort key which is a timestamp. So far, I can only call scan and sort them after they return but eventually I'll want to do pagination and only return ten of the most recent posts.
Hey Stephen!
Based on your description, it sounds like you're trying to use Dynamo as if it were a traditional Relational Database. NoSql databases like Dynamo excel at quick retrieval for known values where you know the key (either singular or composite) that you are looking for. It can be extended to query for a key + a second attribute (sort key), but its functionality is pretty limited.
Concerning your specific problem, you can solve this by having a table setup where your partition key is something like blog id (singular value) and your range/sort key is creation date. The combination of these values should be unique (as you can never create two blog posts at the same time). Using this approach, you can perform a paginated query in descending order on the sort key to get top X results (set your LIMIT to be X on your query). This isn't an ideal setup but should solve your problem.
Hope this helps!
When you say 'sort key', I have been hearing 'sword key' instead, my bad lol