Robots.txt is a text file which is used to instruct the search engines for how to crawl and index the pages on your site. The search engines should be obey the robots.txt. The search engine follow the instruction specified by robots.txt file. If you want to some of your pages are not crawl by search engine, then you can protect that pages in robots.txt. Then the search engine don't crawl particular pages specified by your robots.txt file.
No comments:
Post a Comment