It’s more likely that you think!
So what are Robots and what do they do? We have all heard of spiders and crawlers for search engine optimisation but for a brief explanation please read the following below;
Web indexing robots are used by many search engines such as Google, Yahoo, Bing and others. These web indexing robots are also known as spiders. These spiders/robots are the tools used by engines to harvest data for their search engines. When you submit your website to the engines, you are effectively asking the search engines to send their web indexing robot to your website so that it can be crawled and added to their database.
We recommend robots.txt files as a great way to keep portions of your site off limits to search engine spiders. For instance, there’s probably no reason for Google to index your personal folders/unwanted folders or your cgi-bin directory, so why waste their time and your resources by allowing spiders to access parts of your site you don’t need indexed.
Making sure search engine crawl the correct parts of your website is a very important part of your SEO plan.
