Mark Hensler — Wed, 04 Sep 2002 18:11:13 +0000

Woh, neato!

The ID of this TWF thread is 19021. The URL of that doc includes 1921.

Mark Hensler — Wed, 04 Sep 2002 18:08:14 +0000

Looks like I have something to read for my boring 1-4PM class today.

ROB — Wed, 04 Sep 2002 17:13:44 +0000

hey thanks for the reply Peter, and sorry i didnt acknowledge it sooner.

I ran across this purely by chance last night, The Anatomy of a Large-Scale Hypertextual Web Search Engine which was written by the founders of Google. I found it fascinating, and am now enlightened.

Peter J. Boettcher — Mon, 19 Aug 2002 19:52:54 +0000

I'm not a search engine expert, but I would imagine what the big search engines do, isn't terribly different from what you or I would do to search our site, just taken to the next level. All the data would be stored in efficiently indexed tables (system level) in database farms and the data would be extracted by efficient search algorithms.

They might do stuff like ignore common words (the, and, etc) to improve search speed. They also might not re-execute similar searches, that is, if I logged into Google and searched for "Thai Food", then you log in a few hours later and search for "Thai Food", you wouldn't be searching the whole database again, just getting the results I already fetched (in html). It would only re-execute the query if it is older than a set amount of time (say a day).

As for set-up, that really depends on what you're storing, but a basic set-up could be:

SearchMain table:
SearchMainID (PK Identity)
PageName varchar(50)
PageTitle varchar(50)
PageIndexDate smalldatetime

SearchItem table:
SearchItemID (PK Identity)
ItemMainID (from SearchMain table)
PageContent (text)

I separated into 2 tables to improve performance, there may be times when only the main table needs to be used, by separating the PageContent info into another table performance is kept high (by avoiding the cursor having to scan through a slow text field when it doesn't have too).

Have fun!