You might not want sure internet pages of your web site crawled since they won't be beneficial to customers if located in a search engine's search engine results. Be aware that if your web site uses subdomains and you wish to get certain web pages not crawled on a certain subdomain, you'll need to create a independent robots.Internet site entrepren