Robots.txt is a visual editor for Robot Exclusion Files and a log analyzer software. It allows a user to quickly and easily create the robots.txt files required to instruct search engine spiders, which parts of a Web site are not to be indexed and made searchable by the general Web public and then to identify spiders, which do not keep to those instructions
 
keywords
Robots.txt , site optimization , log file analysis , spider visits , robots.txt editor , spider instructions , search engine indexation , search engine optimization |
