3 Steps Guide to Follow Before Crawling
Ask An SEO Question, Complete Seo Guide, Seo Tools

3 Steps Guide to Follow Before Crawling

Crawl-first SEO has two main parts of the search engine infrastructure. That is crawling and indexing.

If pages aren’t crawled, they can’t be indexed. And if your pages aren’t indexed, they won’t appear in Search Engine Result Pages (SERPs).

This makes crawl-first SEO an important to appear in SERPs.

Crawl-first SEO can:

  • Help you understand how Google crawls sites
  • Identify incompatibilities
  • Help Google to access useful pages
  • Help Google to understand content

Before crawling a site, you make sure you follow this 3-step guide.

  1. Send A Crawl Questionnaire To Client

Here you should ask the following questions:

a> How many do you have on your site?

It’s quite impossible to know the exact number of products in their database or how many of them are provided exactly on online. However, if you can ask the question to your client then you can get the answer.

Knowing how many products the client has, is the most important information for you before crawling. This creates a necessity to conduct a ‘Crawl-first SEO Audit on client site.

You’ve to how many products online before you end the Crawl-first SEO Audit. Here are the two essential questions that you like to answer before ending an audit:

  • Can the crawler access all the product pages on client website? If crawler can’t access, then the best way to find it through the web server logs. This will help you to understand whether search engine bots like Googlebot can access the product page but not your crawler. Otherwise, the reason that cause the problem including JavaScript.


  • Is the crawler accessing more pages than it should? If the crawler you have has excessive product URLs than usual, then it implies a problem with site’s crawl. If there’s a crawler trap then you can find out with your audit.


 b> Do the pages on the site return different content based on user agent?


You need to ask if the content on the pages changes depending on user-agent.

c> Do the product pages return different content depending on country or language?


If the content on the pages changes with Geolocalized IPs or languages then you need to learn that.

d> Are there any crawl blocking accesses on your site?


You need to ask if they’re blocking IPs, user-agents from crawling. Then you would like to learn whether there are any crawl restrictions on site or not.

e> What is server’s bandwidth?

You should explain to the client how many pages per second you can crawl on their site.

If you ask them what the server’s bandwidth is, they’ll fail. Hence, it is recommended to agree with number of pages per second you can crawl on your client’s website.

f> Do you have preferred crawl days or hours?

Clients may have some preferred crawl days or hours. They may want to get their site crawled on weekends or some other time. However, if the client possesses some preferences then SEO audit would take longer hours than expected.

  1. Access & Collect SEO Data

You should ask your client to get access to Google Search Console, Web Analytics Tool, and Web Server Logs.

  1. Follow-Up Search Engine Bots’ Http Headers

As a SEO consultant, you should check what HTTP headers search engine bots request in crawls. If your SEO audit troubles Googlebot, you need to know what HTTP headers Googlebot is requesting from an HTTP server.

It is important as when you say to your clients that you will be crawling their sites as Googlebot crawls then you need to be sure of requesting same HTTP headers as Googlebot from their servers. The response information and data that you collect later from a server depend on what you request in your crawlers in HTTP headers.

Launching a crawl is not an easy job. The aforementioned points are rudimentary to launch a crawl. These 3 steps are needs to be followed before launching the crawl.

Leave a Reply

Your email address will not be published. Required fields are marked *