Re: FN-FORUM: robots.txt

date posted 9th March 2006 12:13

On Thu, 9 Mar 2006, Lorna McArdle - Spinette Designs wrote:

> Am I to understand that the robots text file is something the robots/spiders
> look for when searching the www should I consider adding my own to each site
> so that these robots and spiders can categorise them better.

http://www.robotstxt.org/wc/robots.html

--
Graeme -
Top posting.
What's the most annoying thing about this forum?
Outlook Express: http://home.in.tum.de/~jain/software/oe-quotefix/