https://www.webmaster-forums.net/crss/node/1019341 en https://www.webmaster-forums.net/server-side-scripting/htaccess-ban-list#comment-1115223 <p>ah, didnt realize you were only trying to block the naughty ones. one thing though, the naughty ones probably change theyre agent name regularly also. hell i wouldnt doubt if some of them created a UA name dynamically. Another method to stop all bots is to only allow known legitimate user-agent strings. Unfortunately, crawlers can send any query string they want, but most identify themselves in the agent.</p> Tue, 01 Oct 2002 09:27:08 +0000 ROB comment 1115223 at https://www.webmaster-forums.net https://www.webmaster-forums.net/server-side-scripting/htaccess-ban-list#comment-1115220 <p>Rob, you've got a lot of search engines in your list, I'm just trying to ban the bad ones from leeching email addresses, images and site files.</p> <p>but certain ones like ia_archiver I'll add, it only sucks files and from what I have found through sites has been in legal battles from being aggressive bandwidth theft. </p> <p>Problem with using the robots file, most of these wont read or even look at it, let alone obey it. I've also set up a harvestor trap to get the extra ones I've missed, then can ban them by IP.</p> Tue, 01 Oct 2002 08:57:09 +0000 Busy comment 1115220 at https://www.webmaster-forums.net https://www.webmaster-forums.net/server-side-scripting/htaccess-ban-list#comment-1115218 <p>i should combine mine with yours Busy, looks like you got some i didnt, and vice versa.</p> <p>I can't use the robots.txt directives in my application, but you certainly can with your webserver. Create a file called robots.txt in your web document root and include the lines (not positive on this, from memory):</p> <p>User-Agent: *<br /> Disallow: /</p> <p>And voila, any crawlers (that respect the robot exclusion directives) will skip your site.</p> Tue, 01 Oct 2002 08:02:52 +0000 ROB comment 1115218 at https://www.webmaster-forums.net https://www.webmaster-forums.net/server-side-scripting/htaccess-ban-list#comment-1115217 <p>heh, how strange. i spent the last 4 hours putting together a 'robot agent keyword' list for some stat tracking sofware im working on. I seriously just finished (or gave up) and come here to find a thread on the subject, very wierd (are you reading my mind?)</p> <p>Anyways, here's my list. Note i converted everything to lowercase (because i store useragents in lowercase) and the existence of any of these strings (case-insensitive) in a user-agent should, in theory, indicate a crawler. this is all un-tested also as i just finished this list.</p> <p>aitcsrobot<br /> ao/a-t.idrg<br /> arachnoidea<br /> architextspider<br /> atomz<br /> auresys<br /> awapclient<br /> axis<br /> backrub<br /> bayspider<br /> big brother<br /> bjaaland<br /> black widow<br /> blackwidow<br /> borg-bot<br /> bspider<br /> cactvs chemistry spider<br /> calif<br /> cern-linemode<br /> checkbot<br /> christcrawler<br /> cienciaficcion<br /> combine<br /> computingsite<br /> conceptbot<br /> coolbot<br /> cosmos<br /> crawler<br /> crawlpaper<br /> cusco<br /> customcrawl<br /> cyberpilot<br /> cyberspyder<br /> deweb<br /> diagem<br /> die blinde kuh<br /> dienstspider<br /> digger<br /> digimarc<br /> digimarc cgireader<br /> diibot<br /> dittospyder<br /> dlw3robot<br /> dnabot<br /> dragonbot<br /> duppies<br /> ebiness<br /> eit-link-verifier<br /> elfinbot<br /> emc spider<br /> esirover<br /> esismartspider<br /> esther<br /> evliya celebi<br /> explorersearch<br /> fast-webcrawler<br /> fastcrawler<br /> fdse<br /> felix<br /> fido<br /> fish-search<br /> flipper<br /> folio<br /> foobar<br /> fouineur<br /> freecrawl<br /> funnelweb<br /> g.r.a.b.<br /> gammaspider<br /> gazz<br /> gcreep<br /> gestalticonoclast<br /> getterroboplus<br /> geturl.rexx<br /> glimpse<br /> golem<br /> googlebot<br /> grabber<br /> griffon<br /> gromit<br /> grub-client<br /> gulliver<br /> gulper<br /> gulper web bot<br /> harvest<br /> havindex<br /> hazel's ferret web hopper<br /> hku www robot<br /> hometown spider pro<br /> htdig<br /> htmlgobble<br /> hämähäkki<br /> i robot<br /> ia_archiver<br /> iagent<br /> iajabot<br /> ibm_planetwide<br /> image.kapsi.net<br /> imagelock<br /> imagescape<br /> incywincy<br /> industry canada bot<br /> informant<br /> infoseek<br /> infospiders<br /> ingrid<br /> inspectorwww<br /> internet cruiser<br /> iron33<br /> ispi<br /> israelisearch<br /> javabee<br /> jbot<br /> jcrawler<br /> jeeves<br /> jobo<br /> jobot<br /> joebot<br /> jubiirobot<br /> jumpstation<br /> katipo<br /> kdd-explorer<br /> kit-fireball<br /> kit_fireball<br /> ko_yappo<br /> labelgrab<br /> larbin<br /> legs<br /> libertech-rover<br /> linecker<br /> linkidator<br /> linklint<br /> linkscan<br /> linkwalker<br /> lmtaspider<br /> lmtasspider<br /> lockon<br /> logo.gif crawler<br /> lycos<br /> lycos_spider<br /> magpie<br /> mediafox<br /> mercator<br /> merzscope<br /> mindcrawler<br /> moget<br /> momspider<br /> monster<br /> motor<br /> mouse.house<br /> muscatferret<br /> mwdsearch<br /> nec-meshexplorer<br /> nederland.zoek<br /> netcarta<br /> netcarta_webmapper<br /> netmechanic<br /> netscape-catalog-robot<br /> netscoop<br /> newscan-online<br /> nhsewalker<br /> nomad<br /> northstar<br /> occam<br /> open text<br /> openbot<br /> openfind<br /> opilio<br /> orbsearch<br /> packrat<br /> pageboy<br /> parasite<br /> patric<br /> pbwf<br /> pegasus<br /> peregrinator<br /> perlcrawler<br /> pgp-ka<br /> phpdig<br /> piltdownman<br /> pimptrain<br /> pioneer<br /> plumtreewebaccessor<br /> poppi<br /> portalbspider<br /> portaljuice<br /> psbot<br /> pybot<br /> raven<br /> resume robot<br /> rhcs<br /> road runner<br /> robbie<br /> robocrawl<br /> robodude<br /> robofox<br /> robot<br /> robot du crim<br /> robozilla<br /> roverbot<br /> rules<br /> sabic<br /> safetynet<br /> scooter<br /> searchprocess<br /> senrigan<br /> sg-scout<br /> shagseeker<br /> shai'hulud<br /> sharp-info-agent<br /> sidewinder<br /> simbot<br /> site valet<br /> sitetech<br /> sitetech-rover<br /> slcrawler<br /> sleek<br /> slurp<br /> snooper<br /> solbot<br /> spanner<br /> speedy<br /> spider<br /> spiderbot<br /> spiderline<br /> spiderman<br /> spiderview<br /> spyder<br /> squirrel<br /> ssearcher<br /> suke<br /> suntek<br /> superewe<br /> t-rex<br /> tarantula<br /> tarspider<br /> techbot<br /> templeton<br /> teoma_agent1<br /> titan<br /> titin<br /> tlspider<br /> ucsd-crawler<br /> udmsearch<br /> url spider pro<br /> urlck<br /> valkyrie<br /> verticrawl<br /> victoria<br /> vision-search<br /> voyager<br /> vwbot_k<br /> w3crobot<br /> w3index<br /> w3m2<br /> w3mir<br /> w@pspider<br /> wallpaper<br /> web21<br /> webbandit<br /> webcatcher<br /> webcopy<br /> webcrawler<br /> webfetcher<br /> weblayers<br /> weblinker<br /> webmoose<br /> webquest<br /> webreader<br /> webreaper<br /> webs<br /> webvac<br /> webwalk<br /> webwalker<br /> webwatch<br /> webzone<br /> whatuseek_winona<br /> wired-digital-newsbot<br /> wisewire<br /> wlm-<br /> wolp<br /> wwwc<br /> wwwwanderer<br /> xget</p> Tue, 01 Oct 2002 07:55:35 +0000 ROB comment 1115217 at https://www.webmaster-forums.net