Robots/Spiders crash site

You are viewing this site as a guest. Join our community to get your questions answered and share knowledge. Active members may advertise and ask for a website critique.

Sortman posted this at 14:49—16th November 2007.

They have: 1 posts

Joined: Nov 2007

We have a website that that is continually slowing down or crashing. When we check the logs it's always a bot that seems to have caused it. It's even gone down after the googlebot has visited. The site is a real estate site that has around 4000 properties listed. As far as we know the bots are following all the links on the site, including the dynamic pages. We have considered trying to block the IPs but this seems inpractical as there are so many. Can anyone please advise why this would be happening? It is important for the marketing of the site that we have the details pages indexed but are 4000 pages being crawled too many? Thanks in advance for any help ...

Server Management

Greg K posted this at 19:49—16th November 2007.

He has: 1,597 posts

Joined: Nov 2003

See info on here:

http://www.robotstxt.org/faq/prevent.html

You would need to modify the example given to specify the dynamic pages for your site.

I am curious though, in what way is your site "crashing"? and what type of server is it?

-Greg

[This space intentionally left blank]

Cool Geek Supplies: www.ThinkGeek.com

akopayan posted this at 14:56—19th November 2007.

They have: 62 posts

Joined: Nov 2006

Greg K;226404 wrote: See info on here:

http://www.robotstxt.org/faq/prevent.html

You would need to modify the example given to specify the dynamic pages for your site.

I am curious though, in what way is your site "crashing"? and what type of server is it?

-Greg

Thanks for your info. Didn't know about that.

tummy posted this at 15:01—26th November 2007.