Google Removing Crawl Rate Limiter Tool From Search Console via @sejournal, @martinibuster

Google announced that the Googlebot crawl rate limiter will soon disappear from search console The post Google Removing Crawl Rate Limiter Tool From Search Console appeared first on Search Engine Journal.

Nov 27, 2023 - 21:04

0 14

Google Removing Crawl Rate Limiter Tool From Search Console via @sejournal, @martinibuster

The Googlebot Crawl Rate Limiter Tool will soon be removed from search console because it's no longer necessary

Google announced that it is sunsetting the search console crawl rate limiter tool, scheduled to be removed on January 8, 2024, citing improvements to crawling that has essentially made it unnecessary.

Search Console Crawl Rate Limiter Tool

The crawl rate limiter tool was introduced to search console fifteen years ago in 2008. The purpose of the tool was to provide publishers a way to control Googlebot crawling so that it didn’t overwhelm the server.

There was a time when some publishers experienced too much crawling, which could result in the server being unable to server webpages to users.

Enough people complained that Google eventually released the tool within search console.

The impact of the tool was to provide Google with data. According to Google, requests to limit crawling typically took about a day to go into effect and remained in effect for 90 days.

Why Google Is Removing Rate Limiter Tool

The announcement stated that crawling algorithms have reached a state where Googlebot can automatically sense when a server is reaching capacity and take immediate action to slow down the crawl rate.

Furthermore, Google stated that the tool was rarely used and when it was used, the crawl rate was generally set to the lowest setting.

Moving forward, the minimum crawl rate will by default be set to a lower rate similar to what publishers tended to request.

According to the announcement:

“With the deprecation of the crawl limiter tool, we’re also setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits.

This means that we effectively continue honoring the settings that some site owners have set in the past if the Search interest is low, and our crawlers don’t waste the site’s bandwidth.”