Yes. A robots.txt file. It’s your chance to take control and tell web bots (think webcrawlers) how to crawl the pages of your site. The problem is that webcrawlers, if left to their own spidery devices, may end up crawling EVERYTHING on your site.

This might sound potentially, at least on the surface, like a good thing. But it’s not…