hia all
Ok.. I have had http://www.totalsmudge.co.uk for about 4 months and google is a dailey visitor.
Is there a new google bot out there at the mo? because i have had "crawl-66-249-64-35.googlebot.com" and a few of its frends indexing my site since 06.30GMT today. Also "spider2-4.picsearch.com " has been on my site for over 4 days.
Is something happening with the way crawlers are working?
or has my site just been lucky?
It's just my bandwidth is dropping quite quickly.
Any insight would be greatfull
Mullet
Suzanne posted this at 22:35 — 29th September 2004.
She has: 5,512 posts
Joined: Feb 2000
Do you have a robots.txt file? If a spider is staying that long, you may have trapped it by accident (usually a poorly written spider).
love me, love my brain :: iStockphoto portfolio
Busy posted this at 01:06 — 30th September 2004.
He has: 6,157 posts
Joined: May 2001
Not sure if that is google or not, googles main UA is "Googlebot/2.1 (+http://www.google.com/bot.html)" with the IP 64.68.82.xx (from my files)
more info here http://www.robotstxt.org/wc/active/html/googlebot.html
while the googlebot.com does go to google.com there is no mention of that site.
picsearch.com can be a resource hog
<?bhb if(broken){ echo("It wasn't me
"); } ?>
Learn HTML the ez way - EzHTML.net
Some people are like slinkies, they dont really serve any purpose but they still bring a smile to your face when you push them down the stairs ...
mullet posted this at 17:13 — 1st October 2004.
He has: 31 posts
Joined: Jan 2004
hi i have got a text file robots.txt and it is upto date.i'm just wondering, i use .htaccess file for re aranging the url's from .php? to .html pages. it is designed for goole but could it cause a problem with other crawlers?
I must be doing something right as after 4 months im within the top 30 listing on gooogles uk pages.
As it is i just about keeping up with google now!!
well lets get back to it
http://www.totalsmudge.co.uk
http://www.13outlaws.net