Blimey, don’t know if I’m going to make it to the end of this month in terms of bandwidth at incsub.org, Googlebot has absolutely whacked me again with 5.21 Gig of bandwidth used up already!!!
Don’t really get what’s going on, had this problem a month ago too and it went away but no idea why.
It’s not like incsub.org (and subdomains) gets a heck of a lot of google hits either, 3916 as of today for this month… am baffled… s’pose I can’t ask them to hold off, is anyone else having similar issues?
Silly bots. You might be able to reign them in using a robots.txt file
Here’s a link to a handy dandy robots.txt generator
http://www.mcanerin.com/search-engine/robots-txt.htm
Are you sure it’s Googlebot? Could have been a spoof. Doesn’t sound like Google behaviour to me…
Yeh, I was thinking that… who would bother spoofing me that they were googlebot though? Seems an odd thing to do…. but I guess you never know.
Thanks for the link D’Arcy, will whack that up asap… not like I’ve got any ads anywhere either!
Not that there’s anything wrong with having a few ;)
[...] 1. I reckon this has got to be someone pretending to be googlebot for some nasty reason or another… the net result is that I’ve had to use robots.txt to keep googlebot out - which is not something that I want to do but have to do as otherwise it’ll shut the site down. [...]
Hi James, I’ve been battling with the same problem but on a much smaller scale for a week or more now. It’s for a site with a wikimedia installed and the stats show a bot claiming to be googlebot which returns every 24 hours and uses up a slightly larger slice of bandwidth every night. I’ve added a robots.tx file which excludes the entire wiki directory, as well as a google sitemap.xml which only lists the main content pages, but so far no change in behaviour from the bot.