Bloglines: The lover with a sting

In General on 28/6/2006 at 11:06 pm

Have discovered a little more about recent server issues… it seems that Bloglines, when allowed access to my server, is making ~230 parallel requests/second to it… which is naturally, along with all the other requests, killing everything.

Problem is that a very significant percentage of people hosted on the server - edublogs.org users included - are heavily bloglines dependant (as am I).

I’ve contacted them and emailed them but with no joy and as such have to ban them for the moment… but it’d really make my day if a Bloglines engineer could come along and figure how we could fix things up (bearing in mind I’m going to sleep now). Dunno if they’ve ramped up their crawler but it needs sorting soon.

  1. We have the same problem on WordPress.com, except imagine it with 230,000 blogs. They said if we ping them it shouldn’t be as bad, but that seems to have had no effect on the Bloglines DOS behavior.

    Matt 29 June 2006 at 5am

  2. [...] Bloglines is DOSing blog providers. Every other major crawler implements some sort of per-resolved-IP throttle, why can’t Bloglines? Even if there were a way to opt-out of their hundreds of simultanous crawlers descending on your service, it seems to me the default behavior should be to not be harmful, and then work with large providers on a case-by-case basis to increase the concurrency of requests. We don’t have this problem with any other aggregator or crawler, hosted or non-hosted. « Darwin on Web 2.0 [...]

    Photo Matt » Bloglines DOS 29 June 2006 at 5am

  3. I haven’t had problems with the Bloglines crawler (yet), but the Googlebot was hammering my server a little more than I liked. I wrote to them and asked them to slow it down, and they did. I also suggested that they might consider following Yahoo!’s “crawl_delay” extension to the robots.txt file. Maybe we could lobby that with Bloglines, as well.

    Dougal Campbell 29 June 2006 at 8am

  4. I’m gonna submit a request.

    James Farmer 29 June 2006 at 12pm

  5. Same problem with Google. Wrote a letter but didn’t any response yet :(

    Website Monitoring 4 July 2006 at 10pm

  6. Google happened a while back, search my archives for it - they fixed it in the end.

    James 5 July 2006 at 7pm

  7. James,
    Did you get this resolved?

    Andrew 13 July 2006 at 2am

  8. [...] While I’m providing freedback to Bloglines, I’m wondering if they have fixed the overcrawling problem reported by James a while back. [...]

    Changing Way » Blog Archive » Bloglines-lines-lines 13 July 2006 at 2am

  9. Kinda - they may have tweaked something to decrease intensity but still an issue. Nothing official though (i.e. I haven’t been told they’ve done anything)

    James 13 July 2006 at 8pm