Using Newzcrawler.
Hammering my site all day, every day literally every few minutes.
I banned the bredband IP range, but somehow I slipped up.

abuse email is being sent to your IP.


I’d missed a couple of IP’s. They are now blocked.

I have also mailed Google. I checked the log for today. Googlebot has visited 591 times (at the point I got the access log). It grabs no images, it doesn’t go near the guestblock, it just downloads archived posts. Posts going back to February. Seeing as Google does obey robots.txt, my hatred of it has reduced somewhat, but isn’t this just terribly inefficient ?
Their page is very light on code, yet that is the bandwidth they pay for. Their inefficient searching is paid for by site owners. Is search technology still so poor that they have to grab so much and hope they get it all ? Or is it that they don’t care ?
I mailed the log to Google and have asked for their comments as to how normal this visiting behaviour is.

Posted in T2