Ben Tremblay » crawl rate http://bentremblay.com/en The web, what matters, common sense Wed, 25 Jan 2012 14:52:10 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 Google killed it! http://bentremblay.com/en/google-killed-it http://bentremblay.com/en/google-killed-it#comments Wed, 24 Sep 2008 03:31:22 +0000 Ben Tremblay http://www.frogstr.com/?p=132 What a day. I’m running a website with one of my friend and the project is going very very well.  Within three months, we managed to get a steady 500+ uniques everyday and it’s going up everyday. We almost reached 1,000 uniques today and we are quite happy about the results we are getting for not too much work! We recently (2 days ago) switched to a hostgator reseller account at 25$ a month for this project because the database is so big (100,000+ entries), performance and server load is now an issue. So we switched without any major issues and the awesome tech support team at hostgator helped us resolve some minor problems, but today, it went completely crazy!

Google Crawl rate

If you run a website updated very often, you will notice a new option in Google webmaster tools: you will be offered to accelerate the Google Bot crawl speed for your site! This is the exact option:

We’ve detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn’t use too much of your server’s resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below.

A faster crawl will enable us to crawl your site quickly, but may put more load on your server.

How tempting is that? Being indexed faster, the dream of every webmaster! With that fresh new reseller account I decided to turn the faster crawl rate on.

What a bad idea

What a bad idea that was to turn the faster crawl rate on. Note that the website we run is very very heavy on resources, so what happened to my website may not happen to your website if you enable the option. It took a couple of hours before the Google Bot decided to crawl my site at top speed, but when it did, boom! No more website!

I checked my statcounter account around 5pm to notice no new visitors came to the website within the last 30 minutes. That’s really unusual when you get 500-1000 uniques a day, so I typed my domain name to see if there was anything wrong. The result:

500 – Internal server error

The evil 500 internal error! The error that tells you something bad has happened, but we don’t tell you what it is and there’s no way to find out! So I emailed HostGator and received an answer within a couple of minutes. The problem was that all 25 allowed processes were used, so no more request could come in. The rep killed the processes and guess what? 5 minutes later, same thing! Eventually, it went back up and I made sure to check the normal crawl rate.

Be careful

So be careful if you check that option. Make sure your server can handle the evil Google bot!

]]> http://bentremblay.com/en/google-killed-it/feed 26