Re: FN-FORUM: robots.txt

date posted 9th March 2006 11:56

Lorna McArdle - Spinette Designs wrote:
> I look at my web stats on a regular basses and see on every site the
> file name robots.txt, now I have never added this file to a server so
> why is it there and should I add one.
>
> Am I to understand that the robots text file is something the
> robots/spiders look for when searching the www should I consider
> adding my own to each site so that these robots and spiders can
> categorise them better.

Yes, it advises web spiders which files they are allowed to investigate.
See http://en.wikipedia.org/wiki/Robots.txt for more details.

It's important to note that this is just advisory and it's completely
possible for a spider to ignore robots.txt and go wherever it wants.

Dave...

--
Magnum Solutions Ltd
Open Source Consultancy, Development and Training