greg — Mon, 23 Jul 2007 22:56:12 +0000

ok thanks

I have the full site password protected for now while its in testing stage
after that I will just block out the bad bots

Busy — Mon, 23 Jul 2007 21:36:42 +0000

All bots ignore the robots file at least once, sadly there is no quick and easy .htaccess method (you have to name all the bots).
About the only real thing you can do is use the robots file and password protect the entire site (all pages you'd have to be logged in for)

greg — Mon, 23 Jul 2007 17:28:10 +0000

I thought some robots ignored the robots.txt file?
I was hoping to use .htaccess.

I have the server's public.html folder password protected (which is of course where all the website files are)
will this keep out all robots?

mscreashuns — Sun, 22 Jul 2007 19:34:19 +0000

This is very simple. Create a file in your root directory named robots.txt. Then put the following lines in the file:

Quote:
User-agent: *

Disallow: /

If you simply want to restrict one page, you can either use the above code and specify an exact page instead of a directory, or you can place the following in the actual file:

<META NAME="*" CONTENT="noindex" />