Re: FN-FORUM: robots.txt

date posted 9th March 2006 17:23

Having any sort of robots.txt file will at least stop errors turning up
in your logs, as most web spiders look for it before spidering your
site, so you get errors if you haven't got one.

Don't block your images directory if it might be helping you get
traffic. One of my clients gets a lot of visitors to their site because
of images found in Google Image searches, so now they're using it as
part of their SEO work to help get extra visitors. However, if all of
your images are just site navigation or stock photography, then you may
as well block it as it's probably not helping you much.

Cheers

Paul
On 9 Mar 2006 13:43:08 -0000, "Lorna McArdle - Spinette Designs"
[EMAIL REMOVED] said:
>
> Thanks guys having read a couple of the tutials I can see they aren't
> infact
> what I thought. I did think they could help in SEO's but I see it's more
> about allowing and disallowing access to sites. IE the images folder
> which
> is a good thing to try and disallow.
>
> Regards
> Lorna
> ----- Original Message -----
> From: "Anthony Cartmell" [EMAIL REMOVED]
> To: [EMAIL REMOVED]
> Sent: Thursday, March 09, 2006 1:28 PM
> Subject: Re: FN-FORUM: robots.txt
>
>
> >
> >> Ok then, will the CSS/html sites be enough then, does anyone have a file
> >> I can mirror, obviously if you do then please email off list.
> >
> > An empty file means "you can crawl this site, whoever you are", you don't
> > need any contents.
> >
> > Anthony
--
Paul Silver - Web Development and Technical Search Engine Optimisation
07813 654285 -- http://www.paulsilver.co.uk
Check your web pages: http://www.spidertest.com