Robots.txt file let the search engine crawlers know which URLs should be crawled on your web site. By creating a robots.txt file you are allowing search bots to crawl all or parts of your website.

Htaccess Redirect Generator