marblehost — Fri, 06 Jul 2007 10:39:27 +0000

pr0gr4mm3r wrote: I have my own way for dealing with naughty spiders. I have a directory that's only mentioned in my disallowed section of my .htaccess file. If the spider navigates to it, the IP is blocked.

I don't want block the spider all.I want it go to some specific web page.
Can I use you method let the spider go where I want.

pr0gr4mm3r — Sun, 01 Jul 2007 19:43:43 +0000

I have my own way for dealing with naughty spiders. I have a directory that's only mentioned in my disallowed section of my .htaccess file. If the spider navigates to it, the IP is blocked.

benj — Sun, 01 Jul 2007 16:32:47 +0000

For the links, maybe try a similar approach with the Email Riddler: http://www.dynamicdrive.com/emailriddler/

koitech — Sun, 01 Jul 2007 10:42:56 +0000

this is the only way to stop spidering your links.
but there is another way. you can delete those files from your website.
and there will be no link and no search engine will add those links to their database

this is the best solution i guess

jennbell — Tue, 19 Jun 2007 13:53:06 +0000

Well I ended up just for now, using the robots.txt I am trying to figure anything more out, I want that extra protection, but I dont want to have to use a username and password type thing. I am going to see for about a month what the robots/txt does, and then in the meantime look for anything else that might accompany that to work as well.
I wish it was something more elaborate than that though.

JeevesBond — Tue, 19 Jun 2007 01:13:22 +0000

jennbell wrote: EDIT: Never Mind, I figured something out.

Any chance you could tell us what you did in the end, or is it a secret?

If the search engine spiders are not honouring your robots.txt you could block access using .htaccess, using a username/password. Or you could block using the user agent string. My guess is you did one of those.