How often do you check your crawl stats?

Google Search Console is a great tool full of useful data so that you can understand how your website is performing in search and any issues that Google finds that may be holding it back in search, such as errors and broken schema mark up. However, there is one feature that is a little tucked away and doesn’t seem to get much coverage. This is the Crawl Stats report. It’s under the Settings section in Search Console, which is a little odd, as you can’t actually set anything for crawl stats!

We recently had an issue with a website where new content was not being indexed and some of the old pages were being deindexed. Going through the process of checking that the content was indexable, linked to internally and not blocking search engines, and seeing that everything was fine, we then discovered that the answer lay in the crawl stats report. A recent increase in the “Average Response Time” had decreased the “Total Crawl Requests”. In other words, something was causing Google to take longer to crawl the pages of the website, so it started crawling less often.

We managed to find the cause of the issue, fix it, and crawling and indexing was restored. New content started getting indexed again. It’s good to keep an eye on the Crawl Stats report, as it can highlight an issue which other reports in Search Console will never expose. Google needs to be able to efficiently crawl a website, as fast as possible. It’s a good idea not to make things difficult for GoogleBot!

View More Insights