CptAwesome — Sat, 18 Dec 2004 09:20:13 +0000

Ok, let me clarify a few things:

The spider goes and pulls the pages, if you have all the info on your server, you don't need a spider.

The index is a store of all the raw data, if you have all the info on your server, this is less important.

What does the brunt of the work is a keywords system. For searching the web, a lot of information has to be processed, and pages are ranked in a large database.

When the query is put to the database, by the front end, it goes "Oh, here we go" and sends you the link(s).

On a much smaller scale, you can simply have all your information in a text file, seperated in a logical fashion, and then the end user queries it, and it gets processed.

If you know UNIX/Linux, it's like Grep, on php.net, look up preg_grep and ereg. If you do anything with searching, you'll want to know preg and ereg.

Regular expression resources:
http://regexlib.com/

and preg is similar (but has key differences) to regular expressions.

The second way in which a search engine can work is soley from keywords. This is probably a faster, but not necessarily better solution

After the spider has gone and done it's thing, you now have lets say 1000 pages from the web. Each script can be broken down into its key phrases, say, any word longer than 4 characters? Then there is a database, which has this huge list of keywords, and with each keyword an address to a page. So, the end user, using the front end script, requests "juice" any of the addresses with the keyword of "juice" can be put into an array, and then you can format that back to the enduser.

The google page rank system will just up priority of certain addresses, based on it's own ideals.

So in that case it is all the index, but the index is a lot more complex than you would have first imagined it.