Re: FN-FORUM: robots.txt

date posted 9th March 2006 14:57

> For image folders, I usually have an index.php file that just contains a
> header to the main site.
>
> Does that work for robots? I suspect it does.

Robots only follow links, they don't try to guess directory names (well,
hackers might I suppose, but that's different). So Google will only index
things that are linked to: it's looking at your web pages, not your
directory structure. Robots.txt just says which things not to index, even
though they're linked to.

Anthony
--
www.fonant.com - hand-crafted web sites