Etag : Its a string which represents the version of the requested resource and will change if there is the change in the response data for the same request. Pros : You can use ETag to decide which data should be used , cached data or fresh data from the server.
Where does cached data return from ? In Nodejs with Express lets assume etag is enabled Then from where the cached response being used , is it stored at client side or server side in memory ?
I have used etag by setting eTag property to true on the main web site under IIS, but when file changes, i still get the cached copy not the new one, any help with that please.
@Hussein Nasser , I haven't used etags before. I am considering using it in a use case with the following details. I have a json file stored in a storage account of the cloud provider, it is the responsibility of some micro service in our k8s cluster to read and serve this json (sometime after filtering) to another type of MS in our cluster. All the replicas of the ms read the same file(something known as PVC or persistent storage claim). This enables me to read the file once, store it in memory and in all the following calls to only read the file last modification from metadata to see if I need to read the whole file again. I only want to read again if the file changed.. My question now is if this a good use case for the clients (some other MS servers) and the MSs that serve this file to use etag or LastModified maybe? I think that I can use the last modified date and hash it as etag and it will be the same in all the MSs since its the same shared file.. but I will still need to implement the etag logic in the client servers and maybe use a shared cache between those clients with the etag right? Thanks in advance for your input ;)
Hi Hussein , I was using etag in my work and came across with the doubt that what if we are caching server response in CDN in middle , in this case when browser sends etag with next request[If-None-Match] then this etag will be matched with CDN cached response? or how will it directly goto server for etag matching?
Daniel Lewis if you use web framework like express you can return the etag by setting the header in the response I am not sure you have low level control over etags headers with normal webservers like apache or tomcat
@@hnasr Sorry that was mostly out of my depth, I have a whm with cpanels I have built an AMP site going through the lighthouse report and gtmetrix to try and make it as fast as possible. Its frustrating to me that I have some wordpress sites that are showing as faster than the amp site. Anyway one of the things it said was to use etags and I get the idea, I just do not know how to generate the code so it knows if the resource has changed.
Thanks for explaining ! I have one query -as u said user first req n server will assign e-tag to him for that particular request. - wht if other user request same url as first. Will he get same etag which user 1 got or different? - what is rule of middleware thus middleware leads to cache poisoning even when E-Tag header is using ?
Great questions! That depends on the implementations of the web server some web server generate unique e-tag per document (resource) regardless of the source ip in that case etag will be the same for all users which isn’t a big deal. Some use the source ip or some other mechanisms to generate the etag in that case it will be unique per user. Cache poisoning and HTTP smuggling is a problem with proxies and middle way I talked about them in this video little bit ua-cam.com/video/h1TZpWbucj0/v-deo.html In a nutshell if an attacker can somehow cache a bad website with etag X in the proxy cache layer then a normal user tries to visit a legitimate website that somehow also generate etag X then the user can be served the poisoned cache with bad website.. But its not something simple to get into its tricky Hope that helps
Why can't we use the page/resource hash as a tag? We will not have to deal with the load balancer problem. And we will eliminate the zombie cookies problem too.
You can, some server implementation does that actually. However some implementation uses some other methods to make sure hashing the content everytime doesn’t harm the performance specially if the resource is big or the resource is sitting in a database somewhere and hitting it is costly
I don't know if you have ever used AWS CloudFront but if you have is there anyway to incorporate E-Tags into this service and if so is there any advantage?
thecyrusj13 Hey didnt use AWS Cloud front, but e-tags comes pretty much by default with every web server. Its just a built in feature. You can tell by checking the responses from your requests (in chrome for example) and see if e-tags headers are coming back As for the advantage, it really lower the bandwidth for your consumers and prevent any unnecessary computations on the backend. Thats only true for GET requests though. Hope that helps!
Hi Hussein, Hope you are doing well. Thank you for the last reply on Caching. I want to know why we use e-tag if we already have last modified(if modified since to check freshness of the resource) filed in http response. How e-tag's are calculated.
Excellent question! You use the last modified property on files such as images, text, html etc. However for dynamically generated content such as json response that doesn’t necessarily corresponds to a physical file on disk ( ( e.g. resulted from querying a database )) we use an etag which is a hash of the content could be md5 or SHA. Multiple web servers has different implementation of etags Hope that helpa
@@hnasr Referring to the first cons, for different application servers under a load balancer, if they are configured to use the same algorithm to generate Etag, then it will not be an issue. Is this understanding correct?
Now, if ETags are used as zombie cookies, it will never return an update of the requested page right? Or is the response be configured on the server to send the content for every request regardless of the ETag?
NM vd S excellent question and yes the server code is modified to use eTags as an identity and always sends a fake 304 not modified to the client along a request to recreate cookies if it doesn’t exist. Remember the server is using the eTag only to recreate the cookie. Once the cookie is created we use the cookie as an identity. If the cookie was deleted, the etag is sent and we use that to recreate the cookie. Here is an example of implementation zombie cookie with etags that i did a while back How Un-deletable Zombie Cookies work (with implementation example) ua-cam.com/video/lq6ZimHh-j4/v-deo.html
Jeno yeah most web servers implement them, you just need to make sure you get the same e-tag for the same resource in different stateless web servers. Would be good experiment 🧪 thanks for your comment!
why theher is that drum sound in the background? Tip: you can filter the unwanted sound by sampleling some seconds of video where you dont't say anything and then filtering out those frequencies
Thats for the video, I dont understand most of this etag crap, arent they like a hashed version of the response? For example if an http response object will return with json, then the server will get an etag number out of that? So each time the server needs to check if the etag is a match it has to reload the data to check? Or its cached somewhere or what the fuck?
0:00 Whats an eTag?
4:30 Pros of eTags
7:10 Cons of eTags
12:35 Hulu use of eTags as zombie cookies
Etag : Its a string which represents the version of the requested resource and will change if there is the change in the response data for the same request.
Pros : You can use ETag to decide which data should be used , cached data or fresh data from the server.
Question of the day have you ever used eTags? If not now that you know about them would you consider using them and why?
Where does cached data return from ?
In Nodejs with Express lets assume etag is enabled
Then from where the cached response being used , is it stored at client side or server side in memory ?
I have used etag by setting eTag property to true on the main web site under IIS, but when file changes, i still get the cached copy not the new one, any help with that please.
love your series man ! it is very real based problems which we face in real software develpment ! keep up good work
Thank you! Exactly the explanations I was looking for!
fab gwada glad I could help :)
@Hussein Nasser , I haven't used etags before. I am considering using it in a use case with the following details. I have a json file stored in a storage account of the cloud provider, it is the responsibility of some micro service in our k8s cluster to read and serve this json (sometime after filtering) to another type of MS in our cluster. All the replicas of the ms read the same file(something known as PVC or persistent storage claim). This enables me to read the file once, store it in memory and in all the following calls to only read the file last modification from metadata to see if I need to read the whole file again. I only want to read again if the file changed.. My question now is if this a good use case for the clients (some other MS servers) and the MSs that serve this file to use etag or LastModified maybe? I think that I can use the last modified date and hash it as etag and it will be the same in all the MSs since its the same shared file.. but I will still need to implement the etag logic in the client servers and maybe use a shared cache between those clients with the etag right? Thanks in advance for your input ;)
so what if you visit a different page, then etag tracking doesn't work?
Hi Hussein , I was using etag in my work and came across with the doubt that what if we are caching server response in CDN in middle , in this case when browser sends etag with next request[If-None-Match] then this etag will be matched with CDN cached response? or how will it directly goto server for etag matching?
I get why to use them and I want to I cant seem to figure out how to generate them. Did I miss that part?
Daniel Lewis if you use web framework like express you can return the etag by setting the header in the response
I am not sure you have low level control over etags headers with normal webservers like apache or tomcat
@@hnasr Sorry that was mostly out of my depth, I have a whm with cpanels I have built an AMP site going through the lighthouse report and gtmetrix to try and make it as fast as possible. Its frustrating to me that I have some wordpress sites that are showing as faster than the amp site. Anyway one of the things it said was to use etags and I get the idea, I just do not know how to generate the code so it knows if the resource has changed.
How can we generate an Etag for a ressource PLEASE? in order to add it into a header's request abd avoid 412 HTTP errors? Thanks!
Thanks for explaining ! I have one query
-as u said user first req n server will assign e-tag to him for that particular request.
- wht if other user request same url as first. Will he get same etag which user 1 got or different?
- what is rule of middleware thus middleware leads to cache poisoning even when E-Tag header is using ?
Great questions! That depends on the implementations of the web server some web server generate unique e-tag per document (resource) regardless of the source ip in that case etag will be the same for all users which isn’t a big deal.
Some use the source ip or some other mechanisms to generate the etag in that case it will be unique per user.
Cache poisoning and HTTP smuggling is a problem with proxies and middle way I talked about them in this video little bit ua-cam.com/video/h1TZpWbucj0/v-deo.html
In a nutshell if an attacker can somehow cache a bad website with etag X in the proxy cache layer then a normal user tries to visit a legitimate website that somehow also generate etag X then the user can be served the poisoned cache with bad website.. But its not something simple to get into its tricky
Hope that helps
Why can't we use the page/resource hash as a tag? We will not have to deal with the load balancer problem. And we will eliminate the zombie cookies problem too.
You can, some server implementation does that actually. However some implementation uses some other methods to make sure hashing the content everytime doesn’t harm the performance specially if the resource is big or the resource is sitting in a database somewhere and hitting it is costly
But if Hulu keeps sending you the same e-tag, then the client never sees the new index.html, no?
Correct but they don’t care really if client got a slightly out of date content.
@@hnasr thanks !
I don't know if you have ever used AWS CloudFront but if you have is there anyway to incorporate E-Tags into this service and if so is there any advantage?
thecyrusj13 Hey didnt use AWS Cloud front, but e-tags comes pretty much by default with every web server. Its just a built in feature.
You can tell by checking the responses from your requests (in chrome for example) and see if e-tags headers are coming back
As for the advantage, it really lower the bandwidth for your consumers and prevent any unnecessary computations on the backend. Thats only true for GET requests though. Hope that helps!
Hi Hussein,
Hope you are doing well.
Thank you for the last reply on Caching.
I want to know why we use e-tag if we already have last modified(if modified since to check freshness of the resource) filed in http response.
How e-tag's are calculated.
Excellent question!
You use the last modified property on files such as images, text, html etc.
However for dynamically generated content such as json response that doesn’t necessarily corresponds to a physical file on disk ( ( e.g. resulted from querying a database )) we use an etag which is a hash of the content could be md5 or SHA.
Multiple web servers has different implementation of etags
Hope that helpa
@@hnasr Referring to the first cons, for different application servers under a load balancer, if they are configured to use the same algorithm to generate Etag, then it will not be an issue. Is this understanding correct?
Now, if ETags are used as zombie cookies, it will never return an update of the requested page right? Or is the response be configured on the server to send the content for every request regardless of the ETag?
NM vd S excellent question and yes the server code is modified to use eTags as an identity and always sends a fake 304 not modified to the client along a request to recreate cookies if it doesn’t exist.
Remember the server is using the eTag only to recreate the cookie. Once the cookie is created we use the cookie as an identity. If the cookie was deleted, the etag is sent and we use that to recreate the cookie.
Here is an example of implementation zombie cookie with etags that i did a while back How Un-deletable Zombie Cookies work (with implementation example) ua-cam.com/video/lq6ZimHh-j4/v-deo.html
I will definitely use them. Looks like they are implemented by default in Nginx.
Jeno yeah most web servers implement them, you just need to make sure you get the same e-tag for the same resource in different stateless web servers. Would be good experiment 🧪 thanks for your comment!
Hussein would this be the same be the of CDNs?
Usman Afzal yes this applies to any web server that supports e-tags and CDNa are nothing but web servers.
I was hoping the video to be on-point. :)
What is the alternative of eTag?
building your own client side caching
why theher is that drum sound in the background? Tip: you can filter the unwanted sound by sampleling some seconds of video where you dont't say anything and then filtering out those frequencies
ah ok...that was music...but boy it was anoyting....ahhahaha
Yeah I know that was one bad experiment I no longer use background music. Learned my lesson 😂
Where's the link to the concurrency?
ua-cam.com/video/QCKZ3VZ87Qo/v-deo.html
Done ✔️ thanks for knowledge 🙏
Thats for the video, I dont understand most of this etag crap, arent they like a hashed version of the response? For example if an http response object will return with json, then the server will get an etag number out of that? So each time the server needs to check if the etag is a match it has to reload the data to check? Or its cached somewhere or what the fuck?
please make a video, how http file upload works.
Thanks for the explanation. But this video could have been much shorter. You basically say the same things over and over again
This video could have been just 6 mins instead of 16 mins
Great video
❤️
Thank youuuu
Man your videos are ok but you talk too much! you just make it too long for something can be explained in much shorter time.