AWS re:Invent 2019: [REPEAT 1] Amazon DynamoDB deep dive: Advanced design patterns (DAT403-R1)
Вставка
- Опубліковано 20 сер 2024
- This technical session is for advanced users of Amazon DynamoDB. The patterns and data models discussed in this presentation summarize a collection of implementations and best practices used by large customers-including Amazon retail businesses-to deliver highly scalable solutions for a wide variety of business problems. We delve into strategies for global secondary index sharding and index overloading, scalable graph processing with materialized queries, relational modeling with composite keys, executing transactional workflows on DynamoDB, and more.
![AWS re:Invent 2019: [REPEAT 2] I didn’t know Amazon API Gateway did that (SVS212-R2)](http://i.ytimg.com/vi/yfJZc3sJZ8E/mqdefault.jpg)
![[UA] Team Vitality проти NAVI | IEM Cologne 2024](http://i.ytimg.com/vi/wttO1zVRTC0/mqdefault.jpg)
00:00 Intro
01:00 Agenda list
02:17 Brief History of Data Processing
06:11 Why NoSQL?
07:11 Overview of Amazon DynamoDB
08:43 NoSQL Data Modeling
11:43 Partition Overloading
12:58 Secondary Indexes
14:43 Partition/shard keys in NoSQL
15:14 Write sharding
17:33 Index overloading
19:11 Hot key issue (heat map)
19:48 Get the most out of DynamoDB throughput
20:52 Auto scaling
22:01 Performance at any scale
23:08 Global-scale events: Elastic is the new normal
24:05 Data modeling - It's all about relationships - including NoSQL databases
25:27 SQL vs NoSQL Design Patterns
26:19 Adhoc "joins" in SQL
27:04 Modeled "joins" in NoSQL
31:05 Document vs wide column data modeling
33:31 Indexing efficiently in NoSQL
36:27 Complex queries
39:38 Composite keys
41:30 Modeling relational data
47:26 The Index schema (GSI1)
50:10 The Index schema (GSI2)
51:06 The Index schema (GSI3)
53:17 Design for common patterns
56:28 NoSQL Workbench for DynamoDB: docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.html
57:32 Conclusions
58:22 Learn with AWS Training and Certifications: www.aws.training/
Olá, Fisher Coder Bom dia tem vídeo novo no nosso canal. Não perca!
Fisher why does he say use Relational for OLAP but not oltp isnt it smart to use RDS for betting and banking apps?
This is the first time I've every used playback speed less than 1x
Lol, I was about to go in the comment section to write exactly that
I usually watch the tech talks at 1.25x. This wonderful talk is the only one so far I had to watch 0.75x. :)
Me too, that’s crazy! He really speaks 1.25x. But at 0.75 the audio was a bit unpleasant to hear so I went normal speed 😂
Not sure how many times I've seen this, but EVERYONE needs to see this. "Data is all relational or it doesn't matter", that's just gold.
This is how you give a 3 hour presentation in 59 minutes
This litterally what I thinking.
Watching this again. Relevant and the only reinvent video to watch over and over again. I get something out of it every time I watch it.
Great talk. Watch at 0.75x.
it works
This was beautiful. Watching it for the 3rd time now.
This is a must watch AWS video on DynamoDB. This guy was absolutely awesome!
This session is a must see for anyone working with data and data models.
Using JSON blobs for faster data was a new one for me, at 32:54. The suggestion to use user-provided data like email address rather than opaque UUID for the "id" attribute was also a bit surprising, but makes sense for some use cases. Toward the end, the example tables were too small to read, made it a bit hard for follow along; suggestion: provide slides and also the data models as code (NoSQL Workbench?) so we can follow along at home as we watch again and again. But, as in previous years, Rick's talks are the best things out there for modeling with DynamoDB.
If you want to read the blog post on the JSON object mentioned, it is here. aws.amazon.com/blogs/database/optimizing-amazon-dynamodb-scan-latency-through-schema-design/
🐐 Always amazing talks from Rick
You had 1 hour and you did it! That was a lot to fit into an hour, but very informative.
it was a very right-in-time presentation answering exactly the questions that we had
nosql workbench preview is a great bonus!
The only video i watched .75X speed. Awesome talk
and just after second time I see that video is 59 mins.. Bravo!
Outstanding session! Super useful. Thanks Rick.
Rick always gives great talks, technical and interesting. Thanks!!!
He is talking so fast 🤯
I know some that change the playback speed in settings to .75 or .5?
Kirk Kirkconnell, thank you, 7.5 is good for me
@@NoSQLKnowHow, he speaks so quick that I thought I accidentally changed the speed to 1.25 or even 1.5 )
He is not communicating. 🙄
Great video on designing for your access patterns. However I also feel the need to hear the story about how data is created and updated:
- Do you need to insert multiple records depending on what you are creating?
- I need to be aware that I have to insert my data with GSI_1_PK, GS_1_SK, GS_2_PK ...
- When inserting data in an overloaded index, do I use the convention with a '#' like #floor#04#room#02 or just concatenate the values without any functional information
- If I update my data, do I update in one place or multiple places (probably should try to avoid the latter)
1. You might. The insight behind NoSQL is that storage is cheap and CPU and networking is expensive, so denormalize data to avoid lookups and roundtrips. DynamoDB handles secondary indexes automatically though 2. Have your db model handle this. E.g. derive those attributes from the requisite entity attributes. 3. Depends on if there may be a collision. Use the entity name to disambiguate. 4. Again, denormalization is what makes NoSQL fast and cheap. If your access patterns are relatively stable it shouldn't be an issue.
I loved the talk. thanks it helps me understand more Dynamodb.
2mn into the video and i already now i'm going to learn a lot.
This guy is unbelievable
Great presentation! As a MongoDB guy, now DynamoDB makes more sense. These examples should be in the DynamoDB documentation if not yet
Golden video, thanks!
when a livestock auctioneer is also a dynamo db guru.
Thank you for dark slides.
Write capacity of 10 million ? I had no idea that was even possible..
My brain is overflowed
*Slaps database* Good for 10 million transactions per second!
Fantastic!
How do you determine when to use LSIs versus GSIs? I've seen a lot of talks mention GSIs but AFAIK GSIs also duplicate the tables so now instead of updating a write once, you now pay for two writes. So when is it good practice to determine when to use LSIs vs GSIs and how should we assess the cost implications?
Both LSI's and GSI's require the data to be copied, there is no cost difference between the two. LSI's are limited to using the same Partition Key as the table and therefore are only useful to re-sort or filter the items on the table, not regroup the items. GSI's allow the data to be completely repartitioned so the items can be sorted and grouped on completely different dimensions. LSI's are consistent, GSI's are eventually consistent with P99 latency less than 10ms. LSI's must be defined when the table is created and can never be removed so they are relatively inflexible. In the rare case where strong consistency is an absolute requirement LSI's are the only choice for indexing. If you need to re-group items and strong consistency is required then the Transactions API will allow you to maintain multiple copies of an item in an ACID compliant manner.
Rick Houlihan thanks so much for your reply!
with LSI, you can use consistent read feature, but it's not supported for GSI
If you are looking for the NoSQL Workbench for DynamoDB and want a clickable link, here it is docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.html
The Linux Academy class on DynamoDB Rick mentioned is free, just sign up for a Community account, and can be found at: linuxacademy.com/course/dynamo-db-deep-dive/
I just found this guy works in MongoDB since Dec. 2021.
good bye oracle :)) hello dynamo
I really doubted if I was on 1.25x ot 1.5x playback speed when I watched this.
This is a very good talk - although its a little hard to hear what is being said as he is talking VERY quickly. Note: If you slow down the video to 0.75 it's to slow :)
What about cases when you don’t have something to use as a partition key?
It Looks like dynamodb-datamapper uses uuidv4 by default if you want to autogenerate the partition key.
Maybe type of the item may be a partition key, like "report" or "project".
39:50 to my knowledge, primary key cannot be the same but the last 2 items from bottome have same primary key. why?
Me every 2 minutes: maybe watch it on 1.25 speed instead of 1.5... a second later... wait nvm
12:30
How can we limit the number of orders returned when we query customer metadata + orders + order items together?
Do you have any examples of when DynamoDB is not a good fit?
Almost anything that's OLAP and not OLTP is a poor fit for DynamoDB. If you don't know your usage patterns ahead of time (e.g. ad hoc queries), then use SQL.
top presenter!
He talks about a "feature that allows the user to know partition key usage" around 19:25 mins into the video.
Anyone knows what that feature is? I'm not finding any relevant AWS documentation.
In the video I am showing you a heat map which is a crude version of Cloudwatch Contributor Insights which is now GA for all tables:
docs.aws.amazon.com/amazondynamodb/latest/developerguide/contributorinsights.html
I didn't understand how it works for "Get Meetings - by date and buildings" (44:28) .
For the SK, we have date + building/floor/room, how do we handle this in the code ?
I mean to make a query between date, how is it handled if there is "date|07.106" ?
Do you mean "Get all planned meetings for a certain room within a given time period"?
In this design that would involve getting all meetings within the time period and then filter for your building and room. However, to make querying more efficient you can create a new LSI (PK: buildingId ; SK: floor#room#date) or a new GSI (PK: buildingId#floor#room, SK: date).
If Norm MacDonald went into IT.
How does this have so few views when I have watched it so many times?
Where can we find the presentation deck?
aws.amazon.com/events/events-content/?awsf.filter-series=event-series%23reinvent&cards.q=Amazon%2BDynamoDB%2Bdeep%2Bdive&cards.q_operator=AND
d1.awsstatic.com/events/reinvent/2019/REPEAT_1_Amazon_DynamoDB_deep_dive_Advanced_design_patterns_DAT403-R1.pdf
Does anyone know where I can find the blog post he mentions on wide column vs document part? He mentions it at 33:28
aws.amazon.com/blogs/database/optimizing-amazon-dynamodb-scan-latency-through-schema-design/
57:34 epic 😂
F1 presentation 😊
Slow down champ. Take a breath
The guy talks like an auctioneer
"I work for AWS. DynamoDB is awesome!"
No way!!!
*Gives credit to the the storage technology for things that actually depend on the company's pricing strategy* :/
*No consideration for trade-offs and drawbacks*
*Soft balls*
*General disingenuity and dog and pony show*
This presentation is embarrassing. The technical value is great but the pace of the speaker is clearly too fast.
There are courses that a speaker can do to learn how to speak in public...
I disagree. It is definitely high bandwidth but this is on a track for people that have deep understanding of datastores and the various trade-offs In video form you can slow it down. Not being some elitist as there's plenty of things I watch and are blown away and need to slow it down For a person that is obviously at the top of his game developing a DB that powers the world's biggest online retailer and cloud service he is an excellent presenter.
Your comment is what’s embarrassing. The title of the talk clearly says ADVANCED and the speaker is incredibly experienced. If you’re looking for hand holding, hello world stuff find a different video. If the pace is the issue surely you know you can change the playback speed. 🙄