Googlebot Scheduler: Is Google Overloading Your Server?

As a search engine optimization consultant, my job is to build sites that Google loves to crawl, index, and rank well in the search engine results pages (SERPs).

But what do I do if my SEO efforts are so successful that Googlebot loves my site too much? What happens when Google's crawler robot spikes my server load so high that I get load warnings from my webhosting provider?

That's the problem that led me to develop Googlebot Scheduler.

The Googlebot server load problem

Googlebot can be a mixed blessing. You want Google to crawl your site, but sometimes Google gets carried away and becomes a resource hog. From an SEO perspective, this seems like a good problem to have. After all, this means Google loves your site.

But if Google is over-crawling your site on a shared hosting account, you can quickly go above your load limit and get your account suspended. That's not good! That's why I wrote Googlebot Scheduler.

(NOTE: If you have a site that Google is NOT crawling heavily, maybe you need some SEO consulting help. ;-)

  

What about Google's Webmaster Tools?

Yes, I know that Google Webmaster's Tools allows you to reduce your crawl rate from Normal to Slower for a fixed period of time, but there are two problem's I've experienced with this:

How Googlebot Scheduler solves the server load problem

My Googlebot Scheduler script was born of necessity for one of my own sites with over 33,000 URLs that Google loves to death. I needed a way to limit Googlebot's daily access so it could only crawl within a specific window of time each day, thus reducing the load Googlebot was putting on my server.

There are two easy time settings to make in the script: The allowed time and the disallowed time. When activated on your server through a cron job (or cron simulator, like JCron Scheduler for Joomla), Googlebot Scheduler will change the contents of your robots.txt file at the times specified to either turn on Google's access or turn it off. 

Can You Use Googlebot Scheduler to regulate other Robots?

Absolutely!

You can easily use Googlebot Scheduler to limit the access of other robots through your robots.txt file.

The only reason I've named this script Googlebot Scheduler is that Googlebot seems to be the worst (best?) offender when it comes to drinking bandwidth. But if you have other robots overloading your server, all you have to do is enter them as another user agent in your "allowed" and "disallowed" robots.txt files, and they can easily be controlled.

How Googlebot Scheduler can save you over $1000 in hosting fees per year

Now, you don't have to decide between being Google friendly and running a large website on a shared hosting account. Because you can scale back Googlebot's access to as little as one hour per day, you have complete control over how much bandwidth you give to Google (or any other robot).

This means Googlebot Scheduler is a big money-saver, because Semi-Dedicated Servers can run from $60 to $90 per month on up. And a Dedicated Server can be more than that.

With Googlebot Scheduler, you won't be forced onto a Virtual Private Server, Semi-Dedicated Server, or Dedicated Server unless your human-generated traffic demands it.

God bless,



Kevin Harper

P.s. - For more information on robtots.txt usage, see:

http://www.robotstxt.org/wc/robots.html

P.p.s. - You can check your Google Webmaster's Tools area to verify if your robots.txt syntax is correct.

P.p.p.s. - If your webhosting plan doesn't allow cron jobs, try ResultsInternet.net. For Joomla sites, you can download the free JCron Scheduler component to simulate cron scheduling.

Images:

The images are downsized due to limited space here. The original dimensions may differ.
Click on the image to open it on a new tab.