Thank you so much for your videos! I’m currently a machine learning engineer trying to cover the computer science theory I never learnt at school. These videos are a goldmine!
It's the way how memory is implemented. By the time you are past L3, the latency starts being almost as "slow" as main system ram on a memory bus. L1 is usually very expensive and space eating SRAM
This has been done before. Intel did a L4 cache on broadwell for certain C cpu chips using a big external Sram die. The first problem is that Cache is rather expensive. Die cost scale exponentially with die size so a 100mm^2 die is 4x as expensive as a 50mm^2. And modern CPU cores are actually quite small zen 5 is only around 4mm^2 but the on the full zen 5 CCD half the die area is that 40 MBi of L3 cache and the 8MBi of L2. And as an additional problem thanks to the high clock speeds, as the size of the cache increases the latency will increase as well just moving the data from the cache back to the processor. Cache is also quite power hungry even when off so they tend to want to minimise it for consumer platforms, especially if it is going to be idle a lot like a phone or as intel did allowing the whole cache to be powered off on the performance cores. As for why we still don't see them trying with L4 or L5 cache, there are a lot of so called cache unfriendly workloads. These work loads tend to have a few things in common. Low levels of exploitable instruction level parrallelism, high levels of random branchs (especially consecutive branches) and a large random and sparsely accessed dataset. Doing these things tends to result in a processor being unable to effectively speculate ahead or reorder instructions to hide latency leaving it purely at the mercy of the memory subsystem to determine how long the stall will be. Thanks to the random sparse accesses the caches are unlikely to contain the right data as data is being trashed and discarded constantly and is unlikely to be prefetched correctly thanks to the randomness. A bigger cache may allow you to brute force this problem as AMD has proven with the X3D line of chips but if the data set is still sufficently bigger than cache, that hit rate is not improvement, you will see no improvement in performance. And if your new big cache doesn't have enough bandwidth to feed the improvement in hit rates you will also have a problem where the cores end up stalling waiting to the cache to actually get around to servicing their request. Making a cache higher bandwidth makes it bigger and more expensive. This is part of the reason that a lot of modern CPU optimisations and a lot of HPC software optimisation focuses on how to make sure the data and istruction stream is as predicatable and cache friendly as possible.
@@jedijackattack3594 Well explained, I also wonder when cache miss occurs would the latency be just RAM latency or RAM latency + latencies of all the cache levels combined?
@@chetan_naik depends on the access architecture, usually serial and not parallel, which means the latencies would be added. However a main memory access time is far far larger than a register file access
What happens if cache is not available yet in the L3 when another core tries to access it? Does it wait for it to propagate? Or does it take the last known (old) state?
Thank you so much for your videos! I’m currently a machine learning engineer trying to cover the computer science theory I never learnt at school. These videos are a goldmine!
I'm glad it helps you ☺️
Excellent video. Informative without getting bogged down in too many unnecessary details
Thanks! It's really nice to hear.
You have been subscribed.
Just don't stop making this quality videos
I won't ☺️
Love the series so far, keep it going!
Really good videos. It is really helping me design cache for my Minecraft cpu ❤
Thanks! Good luck
thanks, very clear, I liked & subscribed
Thank you for the support!
Thanks 👍. high quality 💯
תודה
Informative video but why stop at L3 or L4 cache why not add L5, L6, L7 and so on cache to improve performance?
Probably diminishing returns with increased cost and complexity
It's the way how memory is implemented. By the time you are past L3, the latency starts being almost as "slow" as main system ram on a memory bus. L1 is usually very expensive and space eating SRAM
This has been done before. Intel did a L4 cache on broadwell for certain C cpu chips using a big external Sram die.
The first problem is that Cache is rather expensive. Die cost scale exponentially with die size so a 100mm^2 die is 4x as expensive as a 50mm^2. And modern CPU cores are actually quite small zen 5 is only around 4mm^2 but the on the full zen 5 CCD half the die area is that 40 MBi of L3 cache and the 8MBi of L2. And as an additional problem thanks to the high clock speeds, as the size of the cache increases the latency will increase as well just moving the data from the cache back to the processor.
Cache is also quite power hungry even when off so they tend to want to minimise it for consumer platforms, especially if it is going to be idle a lot like a phone or as intel did allowing the whole cache to be powered off on the performance cores.
As for why we still don't see them trying with L4 or L5 cache, there are a lot of so called cache unfriendly workloads. These work loads tend to have a few things in common. Low levels of exploitable instruction level parrallelism, high levels of random branchs (especially consecutive branches) and a large random and sparsely accessed dataset. Doing these things tends to result in a processor being unable to effectively speculate ahead or reorder instructions to hide latency leaving it purely at the mercy of the memory subsystem to determine how long the stall will be. Thanks to the random sparse accesses the caches are unlikely to contain the right data as data is being trashed and discarded constantly and is unlikely to be prefetched correctly thanks to the randomness. A bigger cache may allow you to brute force this problem as AMD has proven with the X3D line of chips but if the data set is still sufficently bigger than cache, that hit rate is not improvement, you will see no improvement in performance. And if your new big cache doesn't have enough bandwidth to feed the improvement in hit rates you will also have a problem where the cores end up stalling waiting to the cache to actually get around to servicing their request. Making a cache higher bandwidth makes it bigger and more expensive.
This is part of the reason that a lot of modern CPU optimisations and a lot of HPC software optimisation focuses on how to make sure the data and istruction stream is as predicatable and cache friendly as possible.
@@jedijackattack3594 Well explained, I also wonder when cache miss occurs would the latency be just RAM latency or RAM latency + latencies of all the cache levels combined?
@@chetan_naik depends on the access architecture, usually serial and not parallel, which means the latencies would be added. However a main memory access time is far far larger than a register file access
You did not explain associativity.
I explained it in the first video of the playlist
very good (and to the point).
Thanks
How did you made this video, cool visuals
Thanks. Photoshop and CapCut
Can I ask where you get this knowledge from? I would like to know more!
My own knowledge as software/firmware engineer + research I do for each video. You can see the sources I used in the description
What happens if cache is not available yet in the L3 when another core tries to access it? Does it wait for it to propagate? Or does it take the last known (old) state?
I didn't fully understand the question. In any case, a cache will never pass stale values to the processor core.
great video. do you have courses in udemy?
No I don't. Should I?
@@BitLemonSoftware YES