Re: FN-FORUM: robots.txt

date posted 9th March 2006 11:57

On Thu, Mar 09, 2006 at 12:32:55PM -0000, Lorna McArdle - Spinette Designs wrote:
>
> I look at my web stats on a regular basses and see on every site the file
> name robots.txt, now I have never added this file to a server so why is it
> there and should I add one.
>
> Am I to understand that the robots text file is something the robots/spiders
> look for when searching the www should I consider adding my own to each site
> so that these robots and spiders can categorise them better.

Yes, you use them to allow/disallow google/... crawlers to index your site.

Here is a tutorial:

http://www.searchengineworld.com/robots/robots_tutorial.htm

This is what I have in mine (not tabbed in in the real file):

User-agent: *
Disallow:

The MOST important thing to note for you Windows users out there is that the file
should follow the Unix line convention -- ie no carriage return at line end.

--
Alain Williams
Parliament Hill Computers Ltd.
Linux Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer.
+44 (0) 787 668 0256 http://www.phcomp.co.uk/

#include