FN-FORUM: robots.txt

date posted 9th March 2006 11:49

I look at my web stats on a regular basses and see on every site the file
name robots.txt, now I have never added this file to a server so why is it
there and should I add one.

Am I to understand that the robots text file is something the robots/spiders
look for when searching the www should I consider adding my own to each site
so that these robots and spiders can categorise them better.

Or am I completely wrong about it's uses and ignore them.

Lorna McArdle
Designer and Artworker
www.spinettedesigns.co.uk
0118 375 3333
07813327809