Googlebot

Behavior

A website will probably be crawled by both Googlebot Desktop and Googlebot Mobile. However, Google transitioned to mobile-first indexing, meaning it predominantly crawls and indexes the web using a smartphone Googlebot. This process began rolling out gradually in 2016 and was completed for all websites by late 2023. The subtype of Googlebot can be identified by looking at the user agent string in the request. However, both crawler types obey the same product token (useent token) in robots.txt, and so a developer cannot selectively target either Googlebot mobile or Googlebot desktop using robots.txt.

Google provides various methods that enable website owners to manage the content displayed in Google's search results. If a webmaster chooses to restrict the information on their site available to a Googlebot, or another spider, they can do so with the appropriate directives in a robots.txt file, or by adding the meta tag <meta name="Googlebot" content="nofollow" /> to the web page. Googlebot requests to web servers are identifiable by a user-agent string containing "Googlebot" and a host address containing "googlebot.com".

Currently, Googlebot follows HREF links and SRC links. There is increasing evidence Googlebot can execute JavaScript and parse content generated by Ajax calls as well. There are many theories regarding how advanced Googlebot's ability is to process JavaScript, with opinions ranging from minimal ability derived from custom interpreters. Since May 2019, Googlebot has used an "evergreen" web rendering service (WRS) that automatically updates to the latest stable version of the Chromium rendering engine, ensuring support for modern browser capabilities such as ECMAScript 6 features. Googlebot discovers pages by harvesting every link on every page that it can find. Unless prohibited by a nofollow-tag, it then follows these links to other web pages. New web pages must be linked to from other known pages on the web in order to be crawled and indexed, or manually submitted by the webmaster.

A problem that webmasters with low-bandwidth web hosting plans have often noted with the Googlebot is that it takes up an enormous amount of bandwidth. This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data. Google provides "Search Console" that allow website owners to throttle the crawl rate.

How often Googlebot will crawl a site depends on the crawl budget. Crawl budget is an estimation of how typically a website is updated. Technically, Googlebot's development team (Crawling and Indexing team) uses several defined terms internally to take over what "crawl budget" stands for.