Written by Ramón Saquete
Table of contents
- 2 How do we test Googlebot’s behaviour?
- 3 Are there any technologies that will stop working with each new update of the Googlebot?
- 4 How does Googlebot evergreen affect polyfills and transpiled code?
- 6 Is AJAX allowed?
- 7 How does Google behave with session variables and client data?
- 8 Conclusions
If we use more advanced features, we should always test them by disabling them in our browser (this can be done from chrome://flags on Chrome), and see how the page behaves in these cases.
In terms of implementation, developers can follow one of these two strategies:
- Progressive enhancement: it consists in starting from a basic content with text and images, and progressively adding more advanced features.
- Graceful degradation: it consists in starting from a content with advanced features, and rolling back to more basic features, until reaching the most basic content.
How do we test Googlebot’s behaviour?
Are there any technologies that will stop working with each new update of the Googlebot?
Sometimes, Google Chrome removes features with its updates (you can stay up to date with them here and here). These are features, which are usually experimental, and disabled by default. For example, WebVR API for virtual reality became obsolete in version 79, to give way to WebXR API for augmented and virtual reality.
But what happens when the feature isn’t that experimental? This is the case of Web Components v0: when Google announced that it was going to remove them with the version 73 (Chrome was the only browser supporting them, and Web Components v1 are supported by all browsers) the “early adopters” requested Google to give them more time to update their code. This delayed their complete removal until February 2020. Nevertheless, it is uncommon for a website to stop rendering correctly as a result of a Googlebot update. Google engineers always try to maintain backward compatibility in their APIs, meaning we usually needn’t worry about this. In any case, if our website implements the onerror event, as we mentioned earlier, we will be able to see in our logs if there is an error.
If we want to be extremely cautious, we can locate the URLs of each template used on our website, and test them all with each Googlebot update, using the mobile friendly test. In any case, it’s good practice to always test the rendering of all templates with each update of our website’s code.
How does Googlebot evergreen affect polyfills and transpiled code?
Having the latest update of the Googlebot engine, it won’t be necessary to load polyfills to get the bot to correctly see the page, but we should do it if we wish that users with incompatible browsers do see it correctly. If we want to support modern browsers only, with Googlebot evergreen we can use less polyfills and the website will load faster.
Be careful with some of these technologies: when they’re used, we should always verify that Googlebot renders them, in case there is still any feature left that hasn’t been implemented by the bot. Moreover, even if it’s rendered, it doesn’t mean that Google will be able to index content displayed inside a 3D or augmented reality image, instead, Googlebot will simply not block its rendering and it won’t return an error.
There are technologies for which Googlebot doesn’t return an error, but it still doesn’t use them for obvious reasons, because Googlebot is not a user, but a crawler. These technologies are:
- Technologies requiring user permissions: for example, when we display a specific type of content, dependent upon a user’s location, through the Navigator.geolocation API. When denied this permission, we will present an error message. This error will get indexed because Googlebot, by default, denies all permission requests, so our best bet is to display a warning and generic content.
- Service Workers: are installed on the browser with the first request, to provide PWA services, such as saving pages on the cache, for offline browsing or push notification purposes. These features make no sense for a crawler. As a consequence, Googlebot simply ignores them.
- WebRTC: it wouldn’t make any sense for the bot to index content of a technology useful for P2P communication between two browsers. Usually, this is used by web applications like Skype or Hangouts once the user has signed in, but Googlebot doesn’t sign in or has video conferences.
Again, this doesn’t mean that Googlebot doesn’t support this technology, it simply doesn’t use it because it doesn’t allow connections.
Is AJAX allowed?
How does Google behave with session variables and client data?
If Googlebot’s behaviour with cookies is based on the fact that with each request it’s like a new user without any cookie, with SessionStorage and other technologies the status stays the same.