This just set off a trap.
http://www.openwebspider.org/
Ignores robots.txt
82.55.69.126 OpenWebSpider/0.3
Bad Bot
I'd tell you but I don't trust you.
I'd tell you but I don't trust you.
This just set off a trap.
http://www.openwebspider.org/
Ignores robots.txt
82.55.69.126 OpenWebSpider/0.3
Does it ignore meta tags, too?
The meta tags don’t actually matter because as yet it would seem that it is not delivering content – it’s just stealing it.
And it IS stealing – go read the webpage.
I have mailed the author about any possible support for robots.txt
Yeah, I browsed over. I still haven’t picked my jaw up off the floor.
No, openwebspider 0.3 (and older versions) doesn’t read robots.txt!
I’m working on it! Version 0.3 is not public yet and I intend to implement robots.txt support as soon as possible, maybe in the next release!
I’m sorry for any problem caused by the spider and thanks for the feedback!
Shen139