Thankgod, yahoo did something about their crawler.
Yahoo! It's been few months now that I've been complaining to them that their crawler was crashing one of my sites. Every night - on the clock @ 2.15am PST – they would unleash 100’s of crawlers – all downloading at the same time! So instead of normal ~100 child process on apache – I would see ~300, 400, 500, 1000 increasing within minutes and then the server would crash… I’ve sent them my logs and politely explained that it’s not very convenient for me to go to the server room in the middle of the night – every night… Anyway, I end up applying mod_evasive on them so now they are still doing this but are limited to ~200 connections so it’s bearable for the server.
-- This message may have been cut off and the rest will only be shown to members. To become a member, click here --