Robots. They are coming to take your content.

I am in the process of revising my site, and discovered for whatever reason that I had an empty robots.txt file present. I know it is only a voluntary ‘standard’, but as far as I know all the major players do respect it. As the overwhelming proportion of users use a search engine that respects the standard, it does form a useful way of shaping what shows up in the general public eye.

I can never remember the syntax though, so for your reference and my recollection – http://www.robotstxt.org

Addendum: I was not familiar with the semi-standard for site maps so I’ve added that as well to see what the effect will be.

Addendum:Ritta Blens has pointed me to another very useful tool for testing the structure of a robots.txt file: https://www.websiteplanet.com/webtools/robots-txt/

Leave a Reply

Your email address will not be published. Required fields are marked *