Do spiders travel in only one direction?
Examplel:
You have two websites (A) and (B). You have a link on (A) to (B), but no link on (B) back to (A)
Because someone out there links to your (A) site, the spider finds its way to your (A) site. It then follows your (A) link to (B). But since you have no link from (B) back to (A), can the spider go back to (A) from (B)?
- aPaddedCell © Copyright 2006-2008 InterMedia. Site and all articles (unless otherwise specified) licensed under a Creative Commons License. Icons based upon the Tango icon set by various artists.
- All trademarks and copyrights on this page are owned by their respective owners. Forum posts are owned by the individual posters.
chrishirst posted this at 21:17 — 12th November 2005.
He has: 379 posts
Joined: Apr 2005
SE Spiders don't travel in any direction at all. They do not crawl links. They are sent out from the datacentre to the page, grab the page source, then are sent to another page which may or may not be on the same site.
Chris
Indifference will be the downfall of mankind, but who cares?
Venue Capacity Monitoring
Code Samples
Henry posted this at 21:37 — 12th November 2005.
They have: 44 posts
Joined: Nov 2005
Chris, please bare with my ingnorance. And I was always under the impression that following links was the way a spider found a site. I have heard it said, and often, that if there are no incoming links to your site that the spider can't find it.
Are you saying that the spider can find your site even if there no incoming links?
Also, I do not understand "datacentre"?
teammatt3 posted this at 02:15 — 13th November 2005.
He has: 1,907 posts
Joined: Sep 2003
Spiders do crawl links, that's how they find your site. Chrishirst is wrong.
And for your question, the spider cannot travel backwards. If you have no link from B to A, A won't get crawled, unless there is another site out there that links to A.
My Site | Regular Expression Tester
Henry posted this at 03:05 — 13th November 2005.
They have: 44 posts
Joined: Nov 2005
Thanks Teammatt. That's what I thought. Just wanted to be sure. Thanks.
demonhale posted this at 05:11 — 13th November 2005.
He has: 3,301 posts
Joined: May 2005
Chrishirst is kinda correct, it goes back to the datacenter, so even though you dont have actual links from other sites, the spider still crawls to find either new sites and Urls,
Yes it does take some weight to the frequency of relative links your site has, but it doesnt work like when if finds a link it will jump to that page... If that is so, then the spider wouldnt finish a site crawl with all the links it needs to jump around...
The search algorithm is fairly responsible the way a bot or spider behaves, so if it takes a code snapshot of your page, it takes note of the links you have but doesnt necessarily jump into them back and forth...
Visit my New Blog Gee, Please... | Designer Graphics 9
bja888 posted this at 05:39 — 13th November 2005.
He has: 902 posts
Joined: Feb 2004
I own optimalsource.bia and optimalsource.net
I advertise with .biz but still get one or 2 .net hits a month. Spider?
Sites hub
Flash Portfolio
OptimalSource.biz
Bompa posted this at 12:56 — 15th November 2005.
He has: 27 posts
Joined: Oct 2003
"crawl links" is a weird term. It sounds like something is "going" somewhere
and if that were true, a "direction" would be necessary.
Most spider bots I have seen work like this.
The bot has a list of URLs. It downloads the source code from the first
then it downloads the source code of the second URL on the list, then
the third URL on the list, etc etc.
With each download, the source code is store for parsing later or on a
different machine. Later, when the parsing takes place, links are taken
from the source code and added to the master list for future downloading.
I hope this does not confuse the issue heh.
Bompa
Get the Edge on your Competitors!
BLACKHAT SEO