Have discovered a little more about recent server issues… it seems that Bloglines, when allowed access to my server, is making ~230 parallel requests/second to it… which is naturally, along with all the other requests, killing everything.
Problem is that a very significant percentage of people hosted on the server – edublogs.org users included – are heavily bloglines dependant (as am I).
I’ve contacted them and emailed them but with no joy and as such have to ban them for the moment… but it’d really make my day if a Bloglines engineer could come along and figure how we could fix things up (bearing in mind I’m going to sleep now). Dunno if they’ve ramped up their crawler but it needs sorting soon.
We have the same problem on WordPress.com, except imagine it with 230,000 blogs. They said if we ping them it shouldn’t be as bad, but that seems to have had no effect on the Bloglines DOS behavior.
I haven’t had problems with the Bloglines crawler (yet), but the Googlebot was hammering my server a little more than I liked. I wrote to them and asked them to slow it down, and they did. I also suggested that they might consider following Yahoo!’s “crawl_delay” extension to the robots.txt file. Maybe we could lobby that with Bloglines, as well.
I’m gonna submit a request.
Same problem with Google. Wrote a letter but didn’t any response yet :(
Google happened a while back, search my archives for it – they fixed it in the end.
James,
Did you get this resolved?
Kinda – they may have tweaked something to decrease intensity but still an issue. Nothing official though (i.e. I haven’t been told they’ve done anything)