the functionality of

arthur — Wed, 22 Dec 2010 06:40:12 +0000

the functionality of robot.txt file is to avoid or to not allow search engine from some of your website pages, this is done whenever if some of your pages are in construction phase, and you dont want search engine to index them.. you can put them there,

robots.txt file can placed in the main directory with index.html page...the same place..

meybe you can try to ban

faca5 — Tue, 21 Dec 2010 13:24:54 +0000

meybe you can try to ban user-agent?!

You can ban all search engine

faca5 — Tue, 07 Dec 2010 14:44:00 +0000

You can ban all search engine using robot.txt. But some search enigine meybe doesnt check your file and index your page.

{links removed}

The robots.txt file is a set

joomlads07 — Sat, 04 Dec 2010 07:08:14 +0000

The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web.

robot.txt files will help you

freddavis — Tue, 26 Oct 2010 18:20:52 +0000

robot.txt files will help you to hide that content that yiu dont want to show to search engines....this file works on all search engines...

will this robots.txt file

johnson.kelly12 — Wed, 28 Apr 2010 07:00:22 +0000

will this robots.txt file work with all search engines

Buy Mobile Phone - Mobile Phone Deals

Usually your site content, if

jakebrown — Fri, 26 Feb 2010 13:05:31 +0000

Usually your site content, if crawled by scrapers and reposted will show up on Google including any links pointing to your site. But Google will usually obey robots.txt and not show your website directly in results.

Robots which do not obey the protocol can still crawl your website. Content scrapers can still work.

ms2134 wrote:I remember a

sequencehosting — Thu, 25 Feb 2010 22:48:45 +0000

ms2134 wrote:

I remember a case a little while ago in which someone did this but there pages still ended up within Googles Index.

This happened because someone else had linked to a page which "Robots, did not allow to be indexed" so it was not indexed; but merly noticed and listed by the Search Engine.

This is true and still happens today. I believe the best way to remove a site is using web master tools.

I have had a problem where

joshbectt — Sun, 10 Jan 2010 01:42:36 +0000

I have had a problem where the robots did still come to my site. But then again, it never really hurt me at all.

I remember a case a little

ms2134 — Fri, 01 Jan 2010 12:02:54 +0000

I remember a case a little while ago in which someone did this but there pages still ended up within Googles Index.

This happened because someone else had linked to a page which "Robots, did not allow to be indexed" so it was not indexed; but merly noticed and listed by the Search Engine.