user agents aren't fool proof, example I often surf around on FF with user agent set as google (is a ff extension), can set it to anything you want, I believe Opera has similar option.
Why do I do it? Because a lot of silly people redirect per browser (usually to a sorry I can't code properly, get ie to view this site type message) so using a no biased user agent like google gets the bare facts without the bull
Back to the question though. whats the point in it? Even if the spider does spider the directory and lists it on search results people aren't going to be able to view the pages, will call you lots of names and go elsewhere. Unless you want to drive people away there is no sanity in it
teammatt3 posted this at 16:44 — 9th September 2006.
Just thought of an example for this. Check out all the indexed pages by WebmasterWorld.com. They are all indexed by Google but when you go to them you have to buy a membership. Notice how they are not cached either.
Busy, since you have user agent string set to Google, can you see the pages when you click on a link from site:webmasterworld.com (go about 10 pages in.) If busy can, then they are redirecting you based on user agent string. If you can't see the pages Busy, then you are being redirected based on IP, which is a TON of work. You would have to gather all the IPs of every single bot for each search engine.
Ya, me to. The user agent switcher FF extension didn't help me get into these pages.
Isn't is illegal to grant search engines access only over the common visitor?
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.
Renegade posted this at 23:28 — 31st July 2006.
He has: 3,022 posts
Joined: Oct 2002
I'm not sure if it's possible or not, but I'm wondering, what's the point in password protecting a file but having it indexed in the first place?
rawkinrich posted this at 12:30 — 2nd August 2006.
They have: 3 posts
Joined: Jul 2006
Just to have some pages indexed in Google, as I'm creating a small site but would like to keep a section away from the public eye.
Sounds silly I know
But I believe it cannot be done anyway.
Renegade posted this at 20:05 — 2nd August 2006.
He has: 3,022 posts
Joined: Oct 2002
Wouldn't having it open for Google be the same as having it open for the public?
rawkinrich posted this at 08:20 — 3rd August 2006.
They have: 3 posts
Joined: Jul 2006
Practically yes, I was just hoping that there were a way to give spiders access.
pr0gr4mm3r posted this at 04:32 — 4th September 2006.
He has: 1,502 posts
Joined: Sep 2006
Actually, what you could do is redirect people away from the part of your site if they aren't a search engine judging by their user agent.
Busy posted this at 10:36 — 4th September 2006.
He has: 6,151 posts
Joined: May 2001
user agents aren't fool proof, example I often surf around on FF with user agent set as google (is a ff extension), can set it to anything you want, I believe Opera has similar option.
Why do I do it? Because a lot of silly people redirect per browser (usually to a sorry I can't code properly, get ie to view this site type message) so using a no biased user agent like google gets the bare facts without the bull
Back to the question though. whats the point in it? Even if the spider does spider the directory and lists it on search results people aren't going to be able to view the pages, will call you lots of names and go elsewhere. Unless you want to drive people away there is no sanity in it
teammatt3 posted this at 16:44 — 9th September 2006.
He has: 2,102 posts
Joined: Sep 2003
Just thought of an example for this. Check out all the indexed pages by WebmasterWorld.com. They are all indexed by Google but when you go to them you have to buy a membership. Notice how they are not cached either.
Busy, since you have user agent string set to Google, can you see the pages when you click on a link from site:webmasterworld.com (go about 10 pages in.) If busy can, then they are redirecting you based on user agent string. If you can't see the pages Busy, then you are being redirected based on IP, which is a TON of work. You would have to gather all the IPs of every single bot for each search engine.
Busy posted this at 21:34 — 9th September 2006.
He has: 6,151 posts
Joined: May 2001
I think it's IP as I've tried accessing those pages before with no luck
pr0gr4mm3r posted this at 21:50 — 9th September 2006.
He has: 1,502 posts
Joined: Sep 2006
Ya, me to. The user agent switcher FF extension didn't help me get into these pages.
Isn't is illegal to grant search engines access only over the common visitor?
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.