The meta tags don’t actually matter because as yet it would seem that it is not delivering content – it’s just stealing it.
And it IS stealing – go read the webpage.
I have mailed the author about any possible support for robots.txt
No, openwebspider 0.3 (and older versions) doesn’t read robots.txt!
I’m working on it! Version 0.3 is not public yet and I intend to implement robots.txt support as soon as possible, maybe in the next release!
I’m sorry for any problem caused by the spider and thanks for the feedback!
Does it ignore meta tags, too?
The meta tags don’t actually matter because as yet it would seem that it is not delivering content – it’s just stealing it.
And it IS stealing – go read the webpage.
I have mailed the author about any possible support for robots.txt
Yeah, I browsed over. I still haven’t picked my jaw up off the floor.
No, openwebspider 0.3 (and older versions) doesn’t read robots.txt!
I’m working on it! Version 0.3 is not public yet and I intend to implement robots.txt support as soon as possible, maybe in the next release!
I’m sorry for any problem caused by the spider and thanks for the feedback!
Shen139