Follow me on twitter
Facebook
Google killed it!
September 23rd, 2008What a day. I’m running a website with one of my friend and the project is going very very well. Within three months, we managed to get a steady 500+ uniques everyday and it’s going up everyday. We almost reached 1,000 uniques today and we are quite happy about the results we are getting for not too much work! We recently (2 days ago) switched to a hostgator reseller account at 25$ a month for this project because the database is so big (100,000+ entries), performance and server load is now an issue. So we switched without any major issues and the awesome tech support team at hostgator helped us resolve some minor problems, but today, it went completely crazy!
Google Crawl rate
If you run a website updated very often, you will notice a new option in Google webmaster tools: you will be offered to accelerate the Google Bot crawl speed for your site! This is the exact option:
We’ve detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn’t use too much of your server’s resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below.
A faster crawl will enable us to crawl your site quickly, but may put more load on your server.
How tempting is that? Being indexed faster, the dream of every webmaster! With that fresh new reseller account I decided to turn the faster crawl rate on.
What a bad idea
What a bad idea that was to turn the faster crawl rate on. Note that the website we run is very very heavy on resources, so what happened to my website may not happen to your website if you enable the option. It took a couple of hours before the Google Bot decided to crawl my site at top speed, but when it did, boom! No more website!
I checked my statcounter account around 5pm to notice no new visitors came to the website within the last 30 minutes. That’s really unusual when you get 500-1000 uniques a day, so I typed my domain name to see if there was anything wrong. The result:
500 – Internal server error
The evil 500 internal error! The error that tells you something bad has happened, but we don’t tell you what it is and there’s no way to find out! So I emailed HostGator and received an answer within a couple of minutes. The problem was that all 25 allowed processes were used, so no more request could come in. The rep killed the processes and guess what? 5 minutes later, same thing! Eventually, it went back up and I made sure to check the normal crawl rate.
Be careful
So be careful if you check that option. Make sure your server can handle the evil Google bot!
23 Responses to “Google killed it!”
What do you think?
Subscribe : A new sketch every morning
Receive this blog in your inbox by entering your email address:
I was about check the feature but desited as i had no knowledge about the minus points. Thanks for the nice post.
Sudama panigrahis last blog post..QUESTION FOR EmPhAsIs: TAG QUESTION
Wow… i guess my blog does not qualify for that option as I am only getting less than 100 unique hits daily at the moment but it is something to watch out for! Cheers!
shaxxs last blog post..How They Landed Here?
very sad. you worked very hard for it and that’s what you got in return. blame it all to them.
thanks for the warning, this could be a great help if i will came along with the same thing. good thing i don’t expect too much visitors for my site. LOL
3 visits a day is not to bad for the traveler
@Sudama: Yes, be careful
@shaxx: In fact for blogs it shouldn’t be a problem because usually there’s no new content every second!
@jigs: hehe I hope you’ll have that problem one day…That will mean your website is popular
first thought is “Wow I want it!”
second thought is “hmm, maybe not”
lol, I’m all confused now! thanks for the tip, hopefully my server can handle it. It gets a few million hits a month now, I’d rather it didn’t go down though because no commentluv for all if that happens!
Andy Baileys last blog post..The beta testers thanks!
This is a good point to consider – one of sites is heavy on resources so I will certainly not be attempting this option.
I guess on the average site though, it shouldn’t make too much difference.
wow, that was really a bad idea. but unexpected…
but why do you want for google to crawl your site fast? if things are going great stick with what you are doing
Hugo Santoss last blog post..Duplicate content penalty
@Andy: haha, if your website can handle a few million hits a month, it should be fine. I think it’s really in the situation where you have a very resources hungry data driven website. Because usually mysql is the bottleneck and requests are queuing up. I love your plugin by the way, it’s great to have you comment on here!
@olly: Totally right, on a blog or something that is not too heavy on resources I don’t think there would be a problem
@Hugo: I mean, it’s like asking a multi-millionaire why he’s still trying to make more money. If the option’s there, why not try?
This is an issue with Hostgator , they recently made some changes on their servers. If you are running a php script that takes up process time , hostgator servers will not be able to handle the request. I ve wasted more than 10 days sorting this with them back and forth on my site ..your process will keep hanging and show you an internal error, you ll have to mail hostgator each time to fix it. I changed my hosting services from host gator for the site to another service provider and I ve had no issue with the site or the script. If it continues you may want to change your hosting services.
I recently added the Webmaster Tolls to my Google account. It’s probably the best thing I’ve ever done for my website!
@ZK: Thanks for clarifying this! I’m very interested in the host you use instead of hostgator…If you can share that with us that would be great!
@Stan: I know, it is a great addition to any website.
Wow, google bot shutting down a server, you should start thinking about optimising your website. Someday a higher traffic could do that instead of the google bot.
Seals last blog post..Balantidiasis
I’ve been optimizing for a while, but there are a couple of things I can’t optimize…You know it has to be resources hungry!
The database is now over 100mb so that’s quite big. Moving to a dedicated server if things continue to grow at that pace will be the solution.
Hi Ben,
I ve switched from HostGator to HostMonster for sites that run on PHP, if the process issue for your site continues and if HostGator cant help restoring it , you can move the site to HostMonster with the same script. The script should run fine with Host Monster
I have the faster crawl rate on a few of my blogs and never had any trouble, feel kinda sorry though!
The only time things wend badly for em was when I suffered the digg effect on Geekywood after that I got several 500 errors and my site was taken down by my host!
A good warning indeed and I thank you for it. I haven’t checked into it myself, but I probably would have if I noticed it.
Dennis Edells last blog post..New Feature! Link Love For Your Thoughts
@ZK: THanks for sharing that info!
@Dennis: No problems for the warning!
Thanks for the tips buddy. Will not even think of it as I don’t want my readers to see CPU exceeded errors. My problem with webmaster tools, though, is that they don’t update the stats very often – usually it is once in 8 or 10 days though they have all data with them.
Cheers,
Ajith
Ajith Edasserys last blog post..10 things that rub your blog visitors the wrong way!
Google… can’t live with it. can’t live without it.
Shared hosting will only take you so far.
Sean.
Seans last blog post..The Best “You Have Mail” Tone. Ever.
[...] talked about the Google bot recently and how the bastard killed one of my website even on a decent reseller account. Seriously, I love the Google bot, it helps me get indexed What [...]
Wow, that’s good information to have. I didn’t even know that one could switch up, but after hearing what happened to you, I’m thinking it might not be such a great idea. Then again, one of my sites only has 175 pages, whereas the other is getting close to 700 pages; not close to 100,000, but one never knows, right?
Mitchs last blog post..$100 Million Dollar Challenge
thanks. Let me ask you, if I use 000web(free hosting) subdomain, will it rank as high as wordpress?
Tired of getting low amounts of useless traffic for your website? Well i want to tell you about a new underground tactic which makes myself $900 every day on 100% AUTOPILOT. I possibly could be here all day and going into detail but why dont you just check their website out? There is a great video that explains everything. So if your seriously interested in producing hassle-free cash this is the site for you. Auto Traffic Avalanche