4 thoughts on “Bad Bot

  1. The meta tags don’t actually matter because as yet it would seem that it is not delivering content – it’s just stealing it.
    And it IS stealing – go read the webpage.

    I have mailed the author about any possible support for robots.txt

  2. No, openwebspider 0.3 (and older versions) doesn’t read robots.txt!
    I’m working on it! Version 0.3 is not public yet and I intend to implement robots.txt support as soon as possible, maybe in the next release!
    I’m sorry for any problem caused by the spider and thanks for the feedback!


Leave a Reply

Your email address will not be published. Required fields are marked *