There is one limitation when using LSI: Your Data being stored within the same partition key isn't anymore allowed to exceed 10GB. If you have reached 10GB you wont be able to add more items (you get the error ItemCollectionSizeLimitExceededException). You can read about this in the aws doc with the title "Local Secondary Indexes" and the subtitle "Item collections in Local Secondary Indexes"
The PartitionKey + SortKey may not be unique, since two subjects in a forum can have same LastPostDateTime. It may not be relevant since it is just an example and the focus is something else, however it can cause confusion in my opinion.
So...let's design and sell a "NoSQL" database, and then add in "fancy" features so that users can pay extra to have a fraction of the SQL features? I genuinely don't understand why you would choose to use a "NoSQL" database for any use-case like this....and I would love to know!
Hi Adam, Thank you for the comment. I have a video on RDS vs DynamoDB that attempts to answer your question. You can watch it here: ua-cam.com/video/crHwekf0gTA/v-deo.html Hope this helps, Daniel
@@BeABetterDev Got it. Also, for GSI, the whole table (data) is physically replicated to multiple machines. Is LSI also copying some portion of data (the data corresponding to the right partition key) and storing them in different orders?
great vid. Thanks. First example on 4:00 mins, first approach scan on entire table, should read all data including forum name EC2 and RDS. You stopped at S3 while not entirely wrong but mildly confusing at first as compared to second approach.
In this example, how can the primary key be assured to be unique? ForumName(Partition key) + LastPostDateTime(SortKey) could definitely be duplicated right?
Thank you for the video. I have a question: once a sort key has been set, a table keeps its data sorted by that key. LSI is considered as an extension of the sort key. How the table gets sorted by a pack of sort key + multiple LSI? I think there is something different behind LSI. Could you shed light on it please?
This is confusing to me as a rookie -- isn't your partitionkey supposed to be unique? Even when using the Sort Key as part of your partition key, there can be duplicates when two forum posts are updated at the same time?
Great video. Keep in mind that more LSI more cost. AWS DOC ==> "A table with many local secondary indexes incurs higher costs for write activity than tables with fewer indexes" However, writing to the base table with 2 LSIs still cost 1 WCU IF you don't write to LSIs as well. If you write to 1 LSI then it costs 2 WCUs ( 1 for the table and 1 for LSI)
You left one of the most important questions unanswered. Does the LSI range key attribute also need to be unique in combination with the hash key (i.e. LSI is just another sort key)
From aws docs: "In a DynamoDB table, the combined partition key value and sort key value for each item must be unique. However, in a local secondary index, the sort key value does not need to be unique for a given partition key value. If there are multiple items in the local secondary index that have the same sort key value, a Query operation returns all of the items that have the same partition key value. In the response, the matching items are not returned in any particular order."
Well explained! I was going through PluralSight and got lost there with LSI and GSI. Had to hop in here as the first and greatfully the last for LSI. GSI would be next. Thanks
Very helpful. A quick question: I have a large JSON data, how can I create a table with that JSON structure. And can we populate the table with a JSON file?
Great videos and explanation, appreciate the effort, tried couple of blogs/documents but very clearly stated here. Can you please give some tutorials on schema design specially some relations?
Thanks for the awesome video. Queries or scans on a local secondary index consume read capacity units from the base table and updates consume write capacity units from the base table. What do you mean by no extra cost with LSI?
It is compared to the GSI, where if you create an another GSI for your table that would incur extra cost. (provision extra wcu/rcu for the table created)
I have a question for scan operation. if I scan by PK and Subject="aaa" for example, does dynamodb scan/read "LastPostDateTime"? or it just scan the specified column (not all columns).
It's a great video but could you explain how Amazon does it internally? How does it not create more tables and still gives such fast results? If we have local index can we choose to not provide it?
thanks for your awesome Video can I ask about the brief difference between GSI and LSI? from the 2 videos I think the difference that LSI must be defined at table creation time, no Extra Cost, max 5 LSI per table, it is alternative to range key while GSI can be defined any time, need extra cost, no limitation per table, also GSI= new Partition key + new sort key am I correct with the above? thanks and best regards
Hi yes you are correct. Also keep in mind the purpose of the LSI and GSI are fundamentally different. LSI helps you define more range-like queries on a different index. Whereas GSI allows you to perform direct key lookups on the corresponding index.
Hey, total newbie to Dynamo here. I have a question regarding combination of GSI and LSI. So, if my table has one GSI and one LSI defined, can I do the query which uses GSI as a hash key value and LSI as a range key value ? That would make my life a lot easier, but I'm not sure if it is possible. Will appreciate any help :)
Very clear! Last limitations section was great. I would also suggest to add what are the drawbacks when should you not use it obviously it does not come for free and you might want to use a different db like a mirror in some other place to do these queries based on your intent there is a reason it's a simple kv. Great video many thanks.
@@BeABetterDev I don't believe this answer is correct. I believe similar to GSIs the LSIs will consume more space and consume WCUs as well. See here docs.aws.amazon.com/amazondynamodb/latest/developerguide/LSI.html where it is mentioned in the AWS documentation "The smaller the index, the less that it costs to store it, and the less your write costs are". This is why there is a trade-off in how many additional attributes you would want to project in your LSI.
Hi Eric, GSIs allow you to query for specific attributes other than the partition key. LSIs allow you to perform 'range-like' queries on your LSI's range key.
Hi Rick, LSIs can unfortuantely only be defined when the table is created and solve a bit of a different purpose than LSIs. LSIs allow you to perform 'range-like' queries on a data field provided you know the partition key. GSIs allow you to perform direct lookups or queries on fields other than the partition key. Hoep this helps!
@@BeABetterDev Hi... First, thank you for this series of AWS videos. Absolutely fantastic!! On the LSI/GSI difference, it caused me to re-read the docs and do some testing. I still have the understanding that a GSI can be built over the same Partition Key (PK); and can use either the same or different Sort key (SK) than the base table, which I believe implies a range. For example: "Select PK='x' and SK between 10 and 15". I believe this represents a "Range-Query" that you mentioned above. If this is true, doesn't that mean the GSI has essentially the same Query capabilities as the LSI? If so, I'm sill not sure what benefit the LSI has over the GSI, other than it's stored along with the base table rather than creating its own Partitions as the GSI does. Thanks again, so much. Rick
There is one limitation when using LSI:
Your Data being stored within the same partition key isn't anymore allowed to exceed 10GB. If you have reached 10GB you wont be able to add more items (you get the error ItemCollectionSizeLimitExceededException).
You can read about this in the aws doc with the title "Local Secondary Indexes" and the subtitle "Item collections in Local Secondary Indexes"
The PartitionKey + SortKey may not be unique, since two subjects in a forum can have same LastPostDateTime. It may not be relevant since it is just an example and the focus is something else, however it can cause confusion in my opinion.
Hi Zacson, thanks for pointing this out. You are absolutely correct.
There you can use the suffix pattern
If you use Unix epoch time it would be unique
@@rongrongmiao3018 Why do you think that the unix epoch time won’t collide when generated at the same time?
So...let's design and sell a "NoSQL" database, and then add in "fancy" features so that users can pay extra to have a fraction of the SQL features?
I genuinely don't understand why you would choose to use a "NoSQL" database for any use-case like this....and I would love to know!
Hi Adam,
Thank you for the comment. I have a video on RDS vs DynamoDB that attempts to answer your question. You can watch it here: ua-cam.com/video/crHwekf0gTA/v-deo.html
Hope this helps,
Daniel
This is exactly what I needed! Thanks very much. Saved me a lot of money.
You are very welcome!
Thank you for your lecture. Is LSI sth like a sort key, without having the uniqueness limitation of (partition key, sort key) pair?
Hi Sunwoo, that's a good way of thinking about it!
@@BeABetterDev Got it. Also, for GSI, the whole table (data) is physically replicated to multiple machines. Is LSI also copying some portion of data (the data corresponding to the right partition key) and storing them in different orders?
One other limitation, just to be clear, is that you can change the sort key, not the partition key in an LSI
great vid. Thanks.
First example on 4:00 mins, first approach scan on entire table, should read all data including forum name EC2 and RDS. You stopped at S3 while not entirely wrong but mildly confusing at first as compared to second approach.
Thanks for the code snippets 7:05 mins (just leaving a shortcut here)
Nice explanation, but I have a little correction. You CAN NOT search by endsWith. only beginwith
Speed: Scan then filter < query on partition key < LSI
Spot on!
In this example, how can the primary key be assured to be unique? ForumName(Partition key) + LastPostDateTime(SortKey) could definitely be duplicated right?
Thank you for the video. I have a question: once a sort key has been set, a table keeps its data sorted by that key. LSI is considered as an extension of the sort key. How the table gets sorted by a pack of sort key + multiple LSI? I think there is something different behind LSI. Could you shed light on it please?
I also have the same doubt. Please explain.
This is confusing to me as a rookie -- isn't your partitionkey supposed to be unique? Even when using the Sort Key as part of your partition key, there can be duplicates when two forum posts are updated at the same time?
Great video. Keep in mind that more LSI more cost. AWS DOC ==> "A table with many local secondary indexes incurs higher costs for write activity than tables with fewer indexes"
However, writing to the base table with 2 LSIs still cost 1 WCU IF you don't write to LSIs as well. If you write to 1 LSI then it costs 2 WCUs ( 1 for the table and 1 for LSI)
Great point! Thanks for sharing.
When would a write to a table not result in a write also to its GSI?
Hi @Robo Profit would be great if you can share any reference please? i have gone through the AWS documentation but couldn't find this. Thanks
@@bradw2k when you don't write to the index and only write to the base table
how can an entire table scan will happen when you have a partition key on forumname in the example for fetching records with s3 and subject = aaa
You left one of the most important questions unanswered. Does the LSI range key attribute also need to be unique in combination with the hash key (i.e. LSI is just another sort key)
From aws docs:
"In a DynamoDB table, the combined partition key value and sort key value for each item must be unique. However, in a local secondary index, the sort key value does not need to be unique for a given partition key value. If there are multiple items in the local secondary index that have the same sort key value, a Query operation returns all of the items that have the same partition key value. In the response, the matching items are not returned in any particular order."
@@renatolimasantos8618 Thanks, I was looking for this additional info. This makes the LSI a lot more useful.
Hi ! Your videos are great! I had trouble understanding the LSI and GSI for a long time. Thank you . going through more of your content.
Thanks so much Mugdha!
Well explained! I was going through PluralSight and got lost there with LSI and GSI. Had to hop in here as the first and greatfully the last for LSI. GSI would be next. Thanks
Thank you so much for exemplifying it so clearly.
You're very welcome Aamer!
Very helpful. A quick question: I have a large JSON data, how can I create a table with that JSON structure. And can we populate the table with a JSON file?
You are the hero who saves me for my presentation of dynamoDB at class , such a great explaination
Great videos and explanation, appreciate the effort, tried couple of blogs/documents but very clearly stated here. Can you please give some tutorials on schema design specially some relations?
Thanks serdar. I'll look into incorporating relations in an upcoming video!
Thanks for the awesome video. Queries or scans on a local secondary index consume read capacity units from the base table and updates consume write capacity units from the base table. What do you mean by no extra cost with LSI?
It is compared to the GSI, where if you create an another GSI for your table that would incur extra cost. (provision extra wcu/rcu for the table created)
Great content. 👍 Thank you.
Thanks Juan! Glad you enjoyed.
Great explanation. I would recommend providing SDK examples with Javascript/NodeJS or Python, instead of Java, though.
I have a question for scan operation. if I scan by PK and Subject="aaa" for example, does dynamodb scan/read "LastPostDateTime"? or it just scan the specified column (not all columns).
Hi Clouder,
Scans will return the entire row, so all the columns.
Hope this helps.
@@BeABetterDev thanks for your effort. keep it up.
Won't LSI also duplicate the data like the global secondary index?
So wouldn't that lead to increased cost ?
same question, if does also create a new table copy like GSI
amazing work buddy
Thank you! Cheers!
It's a great video but could you explain how Amazon does it internally? How does it not create more tables and still gives such fast results? If we have local index can we choose to not provide it?
Hey Mohammad,
Unfortunately there isn't much documentation in terms of how aws structures LSIs so really its anyone's guess.
why is this strongly consistent?
Hello, excellent video! You have any java repository with examples?
Hi Jose, I unfortunately dont but will look into setting one up. Thanks for watching.
thanks for your awesome Video
can I ask about the brief difference between GSI and LSI? from the 2 videos I think the difference that LSI must be defined at table creation time, no Extra Cost, max 5 LSI per table, it is alternative to range key while GSI can be defined any time, need extra cost, no limitation per table, also GSI= new Partition key + new sort key
am I correct with the above?
thanks and best regards
Hi yes you are correct. Also keep in mind the purpose of the LSI and GSI are fundamentally different. LSI helps you define more range-like queries on a different index. Whereas GSI allows you to perform direct key lookups on the corresponding index.
@@BeABetterDev so if looking for a specific index - GSI would be more appropriate?
Hey, total newbie to Dynamo here. I have a question regarding combination of GSI and LSI. So, if my table has one GSI and one LSI defined, can I do the query which uses GSI as a hash key value and LSI as a range key value ? That would make my life a lot easier, but I'm not sure if it is possible. Will appreciate any help :)
You have explained it very smoothly and cleared all the doubts. Thanks
You're very welcome!
Very clear! Last limitations section was great. I would also suggest to add what are the drawbacks when should you not use it obviously it does not come for free and you might want to use a different db like a mirror in some other place to do these queries based on your intent there is a reason it's a simple kv. Great video many thanks.
Great video!
Really well explained. I recommend it!
Awesome
perfect!
Which coding font are you using ?
Liberation Mono
Does LSI doubles the space like GSIs?
Hi Mahbubul,
LSIs do not take extra space.
Thanks for watching!
@@BeABetterDev I don't believe this answer is correct. I believe similar to GSIs the LSIs will consume more space and consume WCUs as well. See here docs.aws.amazon.com/amazondynamodb/latest/developerguide/LSI.html where it is mentioned in the AWS documentation "The smaller the index, the less that it costs to store it, and the less your write costs are". This is why there is a trade-off in how many additional attributes you would want to project in your LSI.
@@The6thProgrammer No more read units but more space consumed I guess?
Well explained ❤️
Thanks Alfredo! Glad you enjoyed :)
Great video. Helped me a lot!!
Glad it helped!
Nice video. What is the use case of GSI and LSI? What is the different between them?
Hi Eric,
GSIs allow you to query for specific attributes other than the partition key. LSIs allow you to perform 'range-like' queries on your LSI's range key.
Should LSI be unique within the same table?
It does not necessarily have to be.
great explanation!
Excellent!!
Thank you! Cheers!
Explained excellently, sir
Thank you
What is the criteria to use an LSI vs. a GSI? Or, is there now no benefit to using LSIs?
Hi Rick,
LSIs can unfortuantely only be defined when the table is created and solve a bit of a different purpose than LSIs. LSIs allow you to perform 'range-like' queries on a data field provided you know the partition key. GSIs allow you to perform direct lookups or queries on fields other than the partition key.
Hoep this helps!
@@BeABetterDev Hi... First, thank you for this series of AWS videos. Absolutely fantastic!!
On the LSI/GSI difference, it caused me to re-read the docs and do some testing. I still have the understanding that a GSI can be built over the same Partition Key (PK); and can use either the same or different Sort key (SK) than the base table, which I believe implies a range. For example: "Select PK='x' and SK between 10 and 15".
I believe this represents a "Range-Query" that you mentioned above. If this is true, doesn't that mean the GSI has essentially the same Query capabilities as the LSI? If so, I'm sill not sure what benefit the LSI has over the GSI, other than it's stored along with the base table rather than creating its own Partitions as the GSI does. Thanks again, so much. Rick
So for LSI, hash key + range key does not need to be the same
partition key needs to be the same right
Thanks, to the point and easy to understand.
Glad it was helpful!
You are great , keep up the excellent wok.
Thank you kindly
take some break when you finish a sentence.
This is the single most useless "database" anyone has ever designed.