Checking stats for my WP pages, I see this in the stats:
I’ve included MSNBot there for comparison. Note the amount of data and the last visit.
Now look what is in my robots.txt:
Now let’s check what the Google help pages say:
“robots.txt is a standard document that can tell Googlebot not to download some or all information from your web server.” ( http://www.google.com/bot.html#robots )