Google Clarifies Why Some Pages Are Crawled But Not Indexed

  • July 3, 2024
  • SEO
No Comments

In May, Gary Illyes from Google participated in an insightful interview at the SERP Conf 2024 conference held in Bulgaria. He addressed questions about the causes of the “crawled but not indexed” issue, offering various explanations that can assist in diagnosing and resolving this common problem.

Although the interview took place in May, it has not received significant attention, with the video being underreported. I came across it thanks to Olesia Korobka (@Giridja), who recently highlighted the interview in a Facebook post.

Despite being months old, the insights shared by Illyes remain relevant and highly valuable for webmasters and SEO professionals.

Understanding the “Crawled – Currently Not Indexed” Notification

The “Crawled – Currently Not Indexed” message in the Google Search Console Page Indexing report indicates that Google has crawled a page but has chosen not to index it. This can be a perplexing issue without clear guidance from Google’s documentation.

During the interview, a question was raised:

“Can ‘crawled but not indexed’ be a result of a page being too similar to other content already indexed?

Is Google suggesting there is enough similar content indexed, and your content is not unique enough?”

Illyes confirmed that one possible reason for this issue is the similarity of the content to other indexed pages. However, he explained that there are several other factors involved.

He elaborated:

“Yes, that can be one interpretation. Ideally, we would divide this category into more specific subcategories, but it is extremely challenging due to the way our data is managed internally.

It could be due to duplicate elimination—where we detect that a page’s content is too similar to existing content in our index, which may have better signals.

However, there are multiple potential causes.”

Impact of Site Quality on Indexing

Gary also mentioned that the overall quality of a website could influence Google’s decision to index pages. He stressed that having a high number of ‘crawled but not indexed’ URLs in the Search Console might indicate broader site quality issues.

He continued:

“The general quality of the site plays a significant role. If the number of ‘crawled but not indexed’ URLs is disproportionately high, it may suggest quality issues with the site.

Since February, I’ve observed cases where we decide to index a large number of URLs from a site because our assessment of the site’s quality has improved.”

Additional Reasons for ‘Crawled Not Indexed’

Illyes also provided other possible explanations for why URLs might be crawled but not indexed, including changes in Google’s perception of a site or technical errors.

He explained:

“One potential reason for seeing an increasing number of ‘crawled but not indexed’ URLs is a change in Google’s perception of the site’s quality.

Another possibility is a technical issue, such as a server error causing the same content to be served to every URL on the site.

There can be many contributing factors.”

Key Takeaways

Gary’s responses offer useful guidance for diagnosing why a web page might be crawled but not indexed by Google:

  • Content similarity to already indexed content
  • Presence of identical content on another site with stronger signals
  • General site quality issues
  • Technical problems

Illyes did not delve deeply into what he meant by ‘better signals’ on another site. However, it likely refers to situations where a site syndicates its content to another site, and Google opts to rank the other site higher.

You can watch Gary’s detailed response at the 9-minute mark in the recorded interview:

Featured Image by Shutterstock/Roman Samborskyi

About BDM

We are a digital marketing firm dedicated to assisting our clients in achieving outstanding outcomes in various crucial sectors.

Request a free quote

We provide expert digital services designed to significantly improve websites' organic search rankings, enabling them to compete effectively for top positions, even with highly competitive keywords.

Subscribe to our newsletter!

More from our blog

See all posts