Is Google’s Frequent Crawling of Your Site a Potential Red Flag?

  • June 19, 2024
  • SEO
No Comments

According to a recent LinkedIn post by Gary Illyes, an Analyst at Google, website owners should exercise caution if they observe that Google is aggressively crawling their website. Although an increase in crawling activity may sometimes signify positive developments, Illyes warns that it could also signal underlying problems that require immediate attention.

Illyes cautions:

“Don’t get happy prematurely when search engines unexpectedly start to crawl like crazy from your site.”

He identifies two prevalent issues that could cause such a spike in crawling activity: infinite spaces and website hacks.

Infinite Spaces Could Cause Crawling Spike

One of the issues highlighted by Illyes concerns websites featuring “infinite spaces.” These could include elements such as calendar modules or product listings with endless filtering options, generating an infinite number of potential URLs.

If a website already experiences substantial crawling, crawlers might become overly enthusiastic about these infinite spaces.

“If your site generally has pages that search users find helpful, crawlers will get excited about these infinite spaces for a time.”

To mitigate this problem, Illyes recommends utilizing the robots.txt file to prevent crawlers from accessing these infinite spaces.

Hacked Sites Can Trigger Crawling Frenzy

Another concerning cause of a sudden increase in crawling activity is a security breach. Hackers might inject spammy or malicious content onto a previously reputable site, initially causing crawlers to interpret it as new content worthy of indexing.

“If a no-good-doer somehow managed to get access…they might flood your site with, well, crap… crawlers will get excited about these new pages for a time and happily crawl them.”

Remain Skeptical Of Crawling Spikes

Illyes advises treating sudden spikes in crawling with suspicion until the root cause is fully understood and addressed.

“Treat unexpected sharp increases in crawling as a symptom…until you can prove otherwise. Or, you know, maybe I’m just a hardline pessimist.”

Fixing Hacked Sites: Help From Google

For sites affected by hacking, Illyes points to a resource that includes a video with further assistance:

Here are the key points to consider:

Tips From Google’s Video

Google’s video provides a step-by-step guide to the recovery process:

1. Identify The Vulnerability

The initial step involves identifying how the hacker managed to gain access. Tools such as Google’s Webmaster Tools can be very useful for detecting these vulnerabilities.

2. Fix The Vulnerability

Once identified, the security loophole must be closed to prevent any future unauthorized access. This may involve updating software, changing passwords, or enhancing other security measures.

3. Clean The Hacked Content

Inspect the entire site’s content and code to remove any spam, malware, defaced pages, or other malicious injections. Security plugins like Wordfence can assist in this cleansing process.

4. Harden Security

In addition to fixing the specific vulnerability, take further steps to strengthen the security of the site. This could involve enabling firewalls, limiting user permissions, and ensuring more frequent updates of software.

5. Request A Review

After closing the vulnerability and removing any hacked content, you can request Google to review the site. Once Google verifies it as clean, any security warnings or blacklists will be removed.

The video notes that the review process for malware issues tends to be quicker (days), whereas for spam issues, it might take longer (weeks) as Google needs to thoroughly inspect the cleanup efforts.

Additional Tips From Google’s John Mueller

Google’s John Mueller has previously shared specific advice on how to recover from the SEO impact of hacked pages:

  1. Use the URL removal tool to quickly deindex the hacked pages.
  2. Focus on improving the overall quality of the site in addition to removing the hacked content.
  3. Understand that lingering impacts may persist for months as the site works to regain Google’s trust.

Why SEJ Cares

Website security is paramount for all businesses, as hacked content can severely impact both trust and search engine rankings.

Google’s Gary Illyes has highlighted that sudden spikes in crawling activity could indicate security breaches or technical issues that need immediate attention for the health of your website.

Featured Image: Stacey Newman/Shutterstock

About BDM

We are a digital marketing firm dedicated to assisting our clients in achieving outstanding outcomes in various crucial sectors.

Request a free quote

We provide expert digital services designed to significantly improve websites' organic search rankings, enabling them to compete effectively for top positions, even with highly competitive keywords.

Subscribe to our newsletter!

More from our blog

See all posts