HTTP cache optimisation

Ramón Saquete

Written by Ramón Saquete

HTTP cache headers allow us to control how each type of file of a website is being cached in a user’s browser. When used correctly, performance will improve considerably for returning users of our site, otherwise, we could be caching indefinitely and hopelessly content and redirects, which in reality, we want to update.

How the HTTP cache works

Caching means storing a duplicate of the original data, for faster access the next time around. The HTTP cache consists in storing files of a website in the browser or an intermediate cache proxy, in a way that the subsequent requests of the same files don’t have to be retrieved from the server of origin. Instead, it is recovered from either the browser cache (which is rather fast, because the request doesn’t need to go through the network); or it gets retrieved from a proxy server (which in theory should be closer, or at least, quicker at serving files, than the server of origin is). This way, the second time around, the loading of all files of the website is done faster, saving server resources.

Cache client
An example of how HTTP cache works in a client browser. The first request reaches the server and returns the entire file. After receiving it, the browser stores it on the cache. The second request retrieves the file directly from the browser’s cache, saving the hassle of another request to the server.

In an intermediate proxy the same files are cached for various clients, so a file requested by one user may not have to be requested to the server for another user, but it’s important to be careful and not end up caching personal data of a user who’ve logged in, as we don’t want to show the data of user A to user B. We can see a visual representation of how this works in the picture below:

Proxy cache
Example of a how a proxy server works. We see that 4 requests are made, and the server only has to respond to 1. If the proxy is server-side (we cannot reach the server without passing by the proxy) it is called inverse proxy. If, on the contrary, it is client-side (the client connects to the Internet through it), it is called direct proxy.

Cache headers for server responses

The HTTP cache is configured with different parameters contributing information to the browser or to the intermediate proxy cache, on how the file must be cached or stored. Said parameters appear inside the protocol’s header, encapsulating the transferred file. We will save on loading time only if we have set up the HTTP header parameters correctly on the server. The configuration can be set up in many ways, we can set it to not request the file to the server when the page is loaded again, or we can configure the parameters to ask the server if the file has changed, and if it hasn’t, the server won’t have to return the file, it will only have to indicate that the file has not been modified with a 304 response code.

Let’s look further into which parameters and configuration options we have for each of them, but if you’re not interested in getting into this amount of detail, you can skip directly to practical use examples.

Cache-control

It is used to define the type of cache policy that is going to be used. It can be applied individually to each file, but more commonly we define a policy for each type of file. The values or rules we are going to use in the responses sent from the server are the as follows:

  • max-age: it’s the maximum time in seconds that the response can be reused by the client without being requested again to the server. For example “max-age=60” means that the file will be reused from the cache for a minute, counting from the moment the request was made. This directive applies to proxies and clients.
  • s-max-age: same as before, but only applied to intermediate proxies.
  • no-store: means that the response is never going to be stored in the client, so it will always be downloaded from the server.
  • no-cache: indicates that it should always check whether the file has been updated in the server, and if it has, then download it again. With this value present, max-age is ignored. The sensible thing would be to use this directive or the previous one (“no-store”), but not both at the same time, although you’ll see on many occasions on some pages both are indeed used at the same time. When this happens, browsers pay attention to the more restrictive directive “no-store”, but the two are set because Internet Explorer 6 used the “no-cache” directive incorrectly, with the meaning of “no-store”. However, this is no longer necessary, as this web browser has rightfully been put to rest.
  • must-revalidate: indicates that we must force the fulfilment of other rules, even if they don’t imply a revalidation. This header exists because the client can be set to apply its own cache policy, such as storing expired requests for some time, or it can return expired responses if the server doesn’t respond. With “must-revalidate” we are telling the client to ignore its configuration and obey the parameters returned from the server. This directive is applied to clients and proxies both.
  • proxy-revalidate: same as the previous one, but only for proxies.
  • public: the response can be cached in intermediate proxy servers, as it does not contain a user’s private information. Proxies usually assume that this is the default option.
  • private: as opposed to the previous value, this one indicates that the response contains private information and must not be stored in an intermediate proxy. We can indicate either “public” or “private”, but it makes no sense to use both at the same time.
  • no-transform: an intermediate proxy should not transform the response, for example, to optimise the compression of an image or to minify a CSS file.

There are more rules, but they are for the client’s cache-control. We’ve already seen here the ones we’re interested in, as these are the ones that can be sent by the server.

ETag

It’s called a validation token. It’s a file hash (hexadecimal number that changes when the file is modified). For example, if the file expired on the cache, or a revalidation policy has been established to always revalidate if it’s updated, the client sends the value ETag to the server inside the parameter “If-None-Match“. And if it matches the one it has stored, it returns a 304 responde code (not modified), otherwise it will send the new file with a new Etag.

Last-Modified

Must contain the last modification date of the file. When the client exists, in a similar manner to the previous parameter, the “If-Modified-Since” header is sent in the request with the latest date it received. If the server has a later date, it will return the file. If it doesn’t, it will return 304.

The existence of Etag and Last-Modified is mandatory for “no-cache”, “max-age”, and “must-revalidate” policies to work, as these parameters are necessary to revalidate the requests. So at least one of them must be there, although they can also be used simultaneously. If there isn’t any kind of validation, it means the full response will always be returned from the server.

Vary

Indicates if the content must be cached independently, based on whether some other parameter varies inside the request header. The values we use will depend on how the website was programmed. For example: “Vary: User-Agent” indicates that if the User-Agent parameter of the client request is different to a previous request (this happens, for example, when we “Request desktop site” in a mobile browser), then the browser will have to cache the returned result as a separate file (this way we cache the mobile and the desktop version). This is necessary on responsive websites, where the HTML generated in the server changes based on the user-agent.

Another example is if we use the value “Cookie” we are saying that the response is different when the client carries a different cookie, which can be applied when the page changes as a result of a cookie change.

We can indicate any HTTP header parameter this way, with the exception of “method”, as only GET method requests are susceptible to being cached. In fact, if we make a request for a cached file using POST, PUT or DELETE methods, it’s invalidated and the request is re-sent.

With this rule it is always recommended to use the “Accept-Encoding” value, to prevent intermediate proxies from returning files with incorrect compression formats to clients not accepting them. For example, if a client accepting the brotli compression format is returned a file in this format, it stores it on the cache. And if the next client only accepts the gzip format, or no compression format at all, but the same file cached in brotli is returned, an error will occur.

Age

This value is only for proxies and it indicates how long the file was in the proxy. It’s necessary to control the time a file spends on the cache, as it needs to count from the moment the server sent it to the proxy, and not from the moment the proxy sent it to the client.

HTTP headers for the old HTTP/1.0 protocol

Clients supporting this protocol probably don’t exist any more. The protocol we more commonly will run into will be HTTP/1.1 or HTTP/2, so presently these headers are not necessary. It’s important to know what they mean, though, in case we do encounter them.

Expires

Expires is the old version of the “Cache-control” parameter, and it only allows to set the date on which the response expires. It’s not the same thing as the “max-age” parameter of “Cache-control”, which sets the maximum time starting from the moment in which the request was made. If the latter one appears, then the “Expires” value is ignored. Setting it to a past date is like saying that it should not be stored on the cache.

Pragma

This header is used with the “no-cache” value to indicate that it shouldn’t be cached in the browser, which is the same thing as “Cache-control” with the “no-store” value.

Practical use examples: what cache headers to apply to each type of file?

Now I am going to explain the optimal parameters for the most common scenarios. For a better fit to a particular case, it is best you review all the available options described in the previous sections, especially with regard to the Vary parameter.

Dynamic files

These are the HTML files we want to be updated as soon as they change, so the header we must set for these files would be:

Cache-Control: no-cache, must-revalidate
Vary: Accept-Encoding
Etag:  "5a0ad72f-1396"
last-modified: Tue, 14 Nov 2017 11:44:47 GMT

Inside Cache-Control we will include the public or private rule, depending on whether the response can be shared with different users or not, respectively.

Another option would be to cache these files for a very limited amount of time, even if they changed. For example, for 5 seconds:

Cache-Control: max-age=5, must-revalidate

This solution is less aggressive, and would economise on resources in situations where the user reloads the page as soon as it’s been downloaded.

Static files

Images (including favicons), JavaScript and CSS files, and fonts must be cached for as long as possible, as these resources rarely change. If we want to force an update of any of these files, we can change their name and reference in the HTML. There are programming tools allowing us to generate a new version number or a hash in the file name automatically, and include it in the code. This is necessary to be able to cache for long periods of time, and synchronise the update of all caches of all files whenever we want. This technique is called fingerprinting, if we use a hash that is a digital fingerprint of the file. It is also called revving, if we use a revision number. When the developer doesn’t use an automatic tool to generate the file name and link it from the code, instead of fingerprinting, what they usually do is add a query string to the URL (for example: style.css?v=1), to avoid having to change the file name and updating the cache all the same. This is the best option, though, as proxy servers do not allow URL caching with query string.

The maximum time we can set is of 1 year.

Cache-Control: public, max-age=31536000
Vary: Accept-Encoding
Etag:  "5a0ad72f-1396"
last-modified: Tue, 14 Nov 2017 11:44:47 GMT

In this case, it won’t matter if the client stores it on the cache for longer than necessary, because if we want to update, we will change the file name, and the must-revalidate rule will not be necessary.

Files that shouldn’t be cached

This situation only affects files containing confidential information about the user. We can use:

Cache-Control: no-cache, no-store, must-revalidate

In this case we don’t have to indicate if private or public storage is allowed, because we are saying that it shouldn’t be cached. For security reasons, here we should be more strict and add “no-cache” to provide IE6 compatibility. It won’t hurt to also add the old “expires” header with a past date and “Pragma: no-cache”.

External domain files

If we include files from external domains, their cache parameters won’t depend on our server, but on the server they are hosted. JavaScript files from Google Analytics, AdWords, and other APIs are files from external domains that should not be cached, as it wouldn’t be practical for all owners of websites using these scripts if they had to change the file name referenced from the HTML each time Google rolls out an update.

When we have scripts that do not get cached, or their cache time is very small, as in this case, automatic website optimisation tools like Google PageSpeed return a warning that there’s an error to be corrected, but the only solution there is to it is to download these files and serve them from our server, and it’s best not to do this, because then we would have a problem as soon as Google updates the code. This is one of those cases where it is best to sacrifice performance in favour of maintainability and reliability of the code.

Avoid caching 301 redirects

301 and 307 redirects are permanent, so Cache-Control parameters are also taken into account. Be careful with this, because if we program a 301 redirect that isn’t correct, and the browser caches it for a year, there won’t be a way whatsoever to force an update on it. So the recommendation here is to not cache 301 redirects (we can simply use Cache-Control: no store, must-revalidate). Besides, we never know if at some point in the future, or after a failed migration we might want to return to the old URLs. If this happens, and we have old redirects cached, programming new ones could cause an infinite redirection loop in the user’s browser, hopping from one URL to another, back to the previous one and again to the next, until the browser returns an error.

302 redirects, however, being temporary, don’t present this issue, so cache parameters are ignored.

Avoid caching 404 errors

404 errors can also get cached, so the recommendation is the same one we’ve given for redirects: it’s best to avoid caching them, because if at some point we want a 404 URL to return results again, we don’t want some clients to keep the error cached.

Conclusion

As we’ve already seen in posts about JavaScript optimisation and CSS optimisation, Web Performance Optimisation is a complex subject, but the HTTP cache is relatively easy to apply, and it improves performance considerably. Other techniques, albeit complex, are necessary to ensure a good user experience, especially on mobile devices, so don’t let that discourage you and keep applying them on your projects.

Additional references

Ramón Saquete
Autor: Ramón Saquete
Web developer at Human Level Communications online marketing agency. He's an expert in WPO, PHP development and MySQL databases.

Leave a comment

Your email address will not be published. Required fields are marked *