Robots/Spiders crash site
We have a website that that is continually slowing down or crashing. When we check the logs it's always a bot that seems to have caused it. It's even gone down after the googlebot has visited. The site is a real estate site that has around 4000 properties listed. As far as we know the bots are following all the links on the site, including the dynamic pages. We have considered trying to block the IPs but this seems inpractical as there are so many. Can anyone please advise why this would be happening? It is important for the marketing of the site that we have the details pages indexed but are 4000 pages being crawled too many? Thanks in advance for any help ...
Greg K posted this at 19:49 — 16th November 2007.
He has: 2,145 posts
Joined: Nov 2003
See info on here:
http://www.robotstxt.org/faq/prevent.html
You would need to modify the example given to specify the dynamic pages for your site.
I am curious though, in what way is your site "crashing"? and what type of server is it?
-Greg
akopayan posted this at 14:56 — 19th November 2007.
They have: 106 posts
Joined: Nov 2006
Thanks for your info. Didn't know about that.
tummy posted this at 15:01 — 26th November 2007.
They have: 121 posts
Joined: Mar 2007
Server crashing is a real calamity, hope you have backed up everything.
shaunx posted this at 11:32 — 30th November 2007.
They have: 8 posts
Joined: Nov 2007
i do faced the same problem tummy..
knorr posted this at 15:31 — 18th December 2007.
They have: 171 posts
Joined: Sep 2006
How do you back up your databases?
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.