Update: A very helpful bloglines engineer got in touch with me and they’ve applied a special script to set up a combined rate limit for 10 requests per second for the edublogs domain. Seems to be working nicely so far so you’re back in the good books Bloglines ;)
A year on Bloglines continues to misbehave attempting to make hundreds to simultaneous connections through it’s crawler to WPMU hosts like edublogs – causing slowness and even crashing even on brand new clusters.
They don’t respond to emails about it and the only solution seems to be to limit their IP (65.214.44.29) to something like 20-40 concurrent connections – naturally massively slowing down the updates that get through.
It stinks and if they don’t do something about it we’re going to actively start asking the 100K+ users of edublogs sites to use Google Reader and discourage their readers /subscribers from using bloglines in favour of another RSS reader.
It’s a real shame as they were pioneers of web based aggregators and I used them for years but to be honest I’ve had enough.
Hello James,
I’m a Bloglines Engineer, and I can help. We do have tools now to rate limit specific hosts down — Please drop me an email and we can help. I’m not sure what happened to emails before to support.. They are supposed to make it our way.
Thanks,
-Paul
I seem to remember Matt complaining about this as well way back when.
I do note that they finally gave that IP address a better rDNS name though.
Paul has been really helpful too – I’ll post about what we’ve worked out once it’s, um, worked out :)