pr0gr4mm3r — Sat, 09 Sep 2006 21:50:29 +0000

Ya, me to. The user agent switcher FF extension didn't help me get into these pages.

Isn't is illegal to grant search engines access only over the common visitor?

Busy — Sat, 09 Sep 2006 21:34:40 +0000

I think it's IP as I've tried accessing those pages before with no luck

teammatt3 — Sat, 09 Sep 2006 16:44:08 +0000

Just thought of an example for this. Check out all the indexed pages by WebmasterWorld.com. They are all indexed by Google but when you go to them you have to buy a membership. Notice how they are not cached either.

Busy, since you have user agent string set to Google, can you see the pages when you click on a link from site:webmasterworld.com (go about 10 pages in.) If busy can, then they are redirecting you based on user agent string. If you can't see the pages Busy, then you are being redirected based on IP, which is a TON of work. You would have to gather all the IPs of every single bot for each search engine.

Busy — Mon, 04 Sep 2006 10:36:56 +0000

user agents aren't fool proof, example I often surf around on FF with user agent set as google (is a ff extension), can set it to anything you want, I believe Opera has similar option.

Why do I do it? Because a lot of silly people redirect per browser (usually to a sorry I can't code properly, get ie to view this site type message) so using a no biased user agent like google gets the bare facts without the bull

Back to the question though. whats the point in it? Even if the spider does spider the directory and lists it on search results people aren't going to be able to view the pages, will call you lots of names and go elsewhere. Unless you want to drive people away there is no sanity in it

pr0gr4mm3r — Mon, 04 Sep 2006 04:32:28 +0000

Actually, what you could do is redirect people away from the part of your site if they aren't a search engine judging by their user agent.

rawkinrich — Thu, 03 Aug 2006 08:20:53 +0000

Practically yes, I was just hoping that there were a way to give spiders access.

Renegade — Wed, 02 Aug 2006 20:05:24 +0000

Wouldn't having it open for Google be the same as having it open for the public?

rawkinrich — Wed, 02 Aug 2006 12:30:21 +0000

Just to have some pages indexed in Google, as I'm creating a small site but would like to keep a section away from the public eye.

Sounds silly I know But I believe it cannot be done anyway.

Renegade — Mon, 31 Jul 2006 23:28:53 +0000

I'm not sure if it's possible or not, but I'm wondering, what's the point in password protecting a file but having it indexed in the first place?