Can Google Crawl and Index Your Site?

LoupeRecently, Google’s Matt Cutts indicated that the biggest SEO mistake he sees is that people do not have crawlable websites. It’s interesting that one of the most common errors is one that is unrelated to mysterious algorithm updates or complex changes in Google’s search methodology. It’s just good old SEO. If you’re guilty of this optimisation error, what can you do to get back on track – and in the SERPs?

If Google cannot crawl and index your site, it cannot categorize it or return it to searchers. The first step is to create, if you have not already, a Google Webmaster Tools accounts. This is going to be the best tool in your toolkit.

In your account, go to the Health section and choose Crawl Stats from the menu on the left side. This will tell you if, and when, Google has crawled your site. While you’re there, also check out Crawl Errors This will provide a report on any broken links, malfunctioning 404s, site errors, and URL errors that prevent Google from crawling.

So, what if you have a soft 404 error, for instance? Webmaster Tools provides thorough, and clear, explanation, as well as instructions on resolving issues.

One more Webmaster Tools feature: Under Health, select Fetch as Google. You can request that the search engine crawl pages that you’ve just uploaded.

To ensure that your site is optimally crawled and index, there are several other steps that you can take:

  • Submit a sitemap. This is just what it sounds like: a map that helps Google identify pages within your website. Luckily, there are free online XML sitemap generators so there is really no excuse not creating and submitting one!
  • Add fresh content. This attracts bots and can help encourage them to keep visiting your site.
  • Put your code on a diet. Lose any bloat and ensure that your code is clean and in compliance with W3C standards. Again, you can find W3C validators online that you can use to help you with this. Lean sites load faster and index more easily.
  • Check for pages that are “disallowed.” Robots.txt files, which request that Google not index certain pages, keeps Google from…well, indexing certain pages. Are any pages “disallowed” that shouldn’t be?

These and other steps can roll out the welcome mat for bots. Make sure Google can crawl and index your site so searchers can find you.

Leave a Comment