Re: FN-FORUM: robots.txt

date posted 9th March 2006 12:31

Ok then, will the CSS/html sites be enough then, does anyone have a file I
can mirror, obviously if you do then please email off list.

Lorna


----- Original Message -----
From: "Anthony Cartmell" [EMAIL REMOVED]
To: [EMAIL REMOVED]
Sent: Thursday, March 09, 2006 1:04 PM
Subject: Re: FN-FORUM: robots.txt


>
> Lorna,
>
>> I look at my web stats on a regular basses and see on every site the file
>> name robots.txt, now I have never added this file to a server so why is
>> it there and should I add one.
>
> To remove the warning message, just create a zero-length (empty) file
> called robots.txt and put in the root folder for the search engines to
> find. You can use robots.txt to indicate whether you would like search
> engines to crawl your site, but since it's advisory rather than compulsory
> to follow the contents it doesn't really help much.
>
> Anthony
> --
> www.fonant.com - hand-crafted web sites
>
> --
> Freelancers, contractors earn more with Prosperity4
> Call 0870 870 4414 or visit www.prosperity4.com
> and benefit from Inland Revenue approved expenses today.
>
> To advertise here: http://www.freelancers.net/advertising.html
>
>