The robots.txt for the domain now only allows bots to index the /wordpress directory and subdirs of that (and a couple of other minor dirs). Good bots are banned from accessing every other directory. Careful – if you go looking at the robots.txt and start poking around where you should not, you could set a trap off, and I will ban every IP that comes through. So why have I banned the bots ?
It’s got little to do with my current attitude to Google, and more to wondering just how my blog (and other directories) benefit the world at large. I don’t believe they do, so why clutter ? I have had the #1 search result for “Mark”, “WordPress help” “XML Parse Error” “Enabled parking” and several others and yes, it’s cool the first time you see your site as #1 (that is a result from within your site – we can all create stupid domain names then go WOW! KOOL! as we get #1 in a field of … 1) but what does it DO exactly ? Apart from bandwidth, very very little. I carry no ads on the hidden pages, I have nothing unique to offer and as I have yet to discover and continue to read a blog found through a search engine, why should I think that being in the G’s results will benefit me then ?
If comments are then considered, I can recall about 6 comments that have been made by what I would call random visitors. Each blog that I read tends to have it’s own circle of commentors and again, why does being on google help this ? I do not see that it does.
So there was this thing that was costing me, yet providing no tangible benefit. Doesn’t make sense to me, so I closed the door.