Busy — Sat, 17 Jul 2004 04:42:10 +0000

MSN is getting to be known as a bad bot, doesn't always read or obey robots.txt and is still only in development stages, the MSNbot has nothing to do with microsoft search. Well it does but the search engine doesnt give it's result to it.
MSN is also a very bad bandwidth hog, it sucks and sucks and sucks, I had them downloading mb's of stuff on a daily basis, downloading stuff like .exe's, .avb's, .zip's etc which have always been restricted in robots but they have no reason to take stuff like that.
Ban via robots.txt first, if they disobey it block them via .htaccess
for robots.txt

User-agent: msnbot
Disallow: /

fifeclub — Sat, 17 Jul 2004 02:36:02 +0000

Busy wrote: MSN (who I recently banned)

Sorry this is off subject but why did you ban MSN? I took a look at the logs on one of my sites and noticed MASSIVE bandwidth usage by msn robots. Is that why? If so, I've got a robots.txt file but what text should I add to it to kick MSN out?

Busy — Fri, 16 Jul 2004 23:09:59 +0000

I use ?id=1234 on one of my sites, has over 2500 variations of numbers and aren't hardcoded anywhere, to get to them they must first pass through an option (how they are displayed) this option also includes an id and sub id (? and &). I did it this way in hope search engines wouldn't follow them but sadly they do.
Google, ask Jeeves, MSN (who I recently banned), Yahoo and others have no problem finding and indexing all of them.
Google isn't to bad as they only do small bunches, ask Jeeves and MSN just go all out and grab as much as they can.

Suzanne — Fri, 16 Jul 2004 15:47:11 +0000

Server-side processing means that the pages are compiled on the server before they are served to any web-client, whether it's a browser or a spider.

However, to answer your question, best practice is to have more-or-less static pages indexed by making them "permanent" pages. So make a page called lastname.php and submit that. You can even automate it to be entirely hard-coded if you want, or use url-mapping instead. This is best practice for search engines AND for humans -- sensible, readable urls.

fifeclub — Fri, 16 Jul 2004 13:30:27 +0000

That's what I thought at first but if the link is generated dynamically on the server side then how would Google know whether it was hard coded or not? Yes some of my dynamic pages are indexed proving that it is at very least possible. That's why I'm edging towards the theory of "dynamic links will get indexed only if they are located on a non-dynamically created page (no "?" in the url)." But that's just a guess.

vexcom — Fri, 16 Jul 2004 05:58:38 +0000

I think it goes by whether or not you have a hard link in your site to that page

widgets

should get indexed