02:43 Operating Storage at Scale 04:32 Insights into your Amazon S3 storage and performance usage 14:53 Optimising costs for various access patterns 22:59 S3 Intelligent Tiering 27:24 Achieving high performance with Amazon S3 36:05 AWS Common Runtime (CRT) 37:30 Mountpoint for Amazan S3 41:05 How Anthropic uses Amazon S3 to train foundation models
31:00 does anyone know what did he mean by s3 learning prefix? so the PUT optimization is not hard formulated but rather being continuously adaptive learning?
It sounds like multiple prefixes that have a base prefix do not get the 3500 Puts and 5500 Gets for each prefix out of the gate but S3 learns and adapts over time if the S3 activity breaches those boundary numbers. The example he gives identifies a prefix naming pattern that S3 does not have to learn on a daily basis.
24:31 - S3IT No it does NOT. What it does do is monitor the date you put it into S3IT, then moves it to S3IA after 30 days, S3GIR after 90 days, and then if you allow it, S3FR and GDA after 180 days. We've seen no evidence that it moves the material based on real access patterns after over a year of usage for a 700TB prefix with 10s of thousands of objects.
Intelligent-Tiering storage class tiers act on the access date of the object. Lifecycle rules act on the creation date of the object. To see the tier that an object is in in Intelligent-Tiering, use S3 Inventory. If you've found you can GET an object that's in the IA or AIA tiers without it moving up to FA, that's not expected so please discuss with your account team!
02:43 Operating Storage at Scale
04:32 Insights into your Amazon S3 storage and performance usage
14:53 Optimising costs for various access patterns
22:59 S3 Intelligent Tiering
27:24 Achieving high performance with Amazon S3
36:05 AWS Common Runtime (CRT)
37:30 Mountpoint for Amazan S3
41:05 How Anthropic uses Amazon S3 to train foundation models
31:00 does anyone know what did he mean by s3 learning prefix? so the PUT optimization is not hard formulated but rather being continuously adaptive learning?
It sounds like multiple prefixes that have a base prefix do not get the 3500 Puts and 5500 Gets for each prefix out of the gate but S3 learns and adapts over time if the S3 activity breaches those boundary numbers. The example he gives identifies a prefix naming pattern that S3 does not have to learn on a daily basis.
24:31 - S3IT No it does NOT. What it does do is monitor the date you put it into S3IT, then moves it to S3IA after 30 days, S3GIR after 90 days, and then if you allow it, S3FR and GDA after 180 days. We've seen no evidence that it moves the material based on real access patterns after over a year of usage for a 700TB prefix with 10s of thousands of objects.
Intelligent-Tiering storage class tiers act on the access date of the object. Lifecycle rules act on the creation date of the object. To see the tier that an object is in in Intelligent-Tiering, use S3 Inventory. If you've found you can GET an object that's in the IA or AIA tiers without it moving up to FA, that's not expected so please discuss with your account team!