Re: FN-FORUM: robots.txt

date posted 9th March 2006 12:22

Lorna,

> I look at my web stats on a regular basses and see on every site the file
> name robots.txt, now I have never added this file to a server so why is
> it there and should I add one.

To remove the warning message, just create a zero-length (empty) file
called robots.txt and put in the root folder for the search engines to
find. You can use robots.txt to indicate whether you would like search
engines to crawl your site, but since it's advisory rather than compulsory
to follow the contents it doesn't really help much.

Anthony
--
www.fonant.com - hand-crafted web sites