Comments on: On the dangers of spidering badly http://williamtozier.com/slurry/2006/10/16/on-the-dangers-of-spidering-badly Pontification without all the gritty gravitas Sun, 20 Jul 2008 00:11:43 +0000 http://wordpress.org/?v=2.5.1 By: Nemo http://williamtozier.com/slurry/2006/10/16/on-the-dangers-of-spidering-badly#comment-14043 Nemo Fri, 29 Dec 2006 08:38:38 +0000 http://williamtozier.com/slurry/2006/10/16/on-the-dangers-of-spidering-badly#comment-14043 No kidding! They're still in action, hitting a server I run for ~900 pages in less than three minutes; I wondered why the load was over 2... <blockquote> server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 4 idle, and 103 total children </blockquote> It's like a one-person ./ing... <i># apf -d 208.101.36.2</i> No kidding! They’re still in action, hitting a server I run for ~900 pages in less than three minutes; I wondered why the load was over 2…

server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 4 idle, and 103 total children

It’s like a one-person ./ing…

# apf -d 208.101.36.2

]]> By: Tozier http://williamtozier.com/slurry/2006/10/16/on-the-dangers-of-spidering-badly#comment-3975 Tozier Tue, 17 Oct 2006 02:39:35 +0000 http://williamtozier.com/slurry/2006/10/16/on-the-dangers-of-spidering-badly#comment-3975 I'm not sure what an appropriate interval is, but in this case the pair.com admin told me there were 315 concurrent blog processes running, since every time a link is clicked one launches, and so many were launched in such a short time that one didn't finish before the next started. I’m not sure what an appropriate interval is, but in this case the pair.com admin told me there were 315 concurrent blog processes running, since every time a link is clicked one launches, and so many were launched in such a short time that one didn’t finish before the next started.

]]> By: Alex http://williamtozier.com/slurry/2006/10/16/on-the-dangers-of-spidering-badly#comment-3974 Alex Tue, 17 Oct 2006 02:06:13 +0000 http://williamtozier.com/slurry/2006/10/16/on-the-dangers-of-spidering-badly#comment-3974 So ... what -is- a good backoff interval [assuming the links you want to follow actually make sense to follow] ? Or, for that matter, why does fast spidering piss off the admin -- because it looks like a DoS attack ? So … what -is- a good backoff interval [assuming the links you want to follow actually make sense to follow] ? Or, for that matter, why does fast spidering piss off the admin — because it looks like a DoS attack ?

]]>