If I have "Disallow: /gallery/" in my robots.txt file, will that stop search engines (who listen to the rules) from indexing all the files and subfolders within that directory? Like /gallery/album5/ or do I need to make seperate disallows for every folder within /gallery/?
Thanks in advance.
Nip it in the bud!
Busy posted this at 21:10 — 23rd January 2005.
He has: 6,157 posts
Joined: May 2001
If the robots reads the robots txt file it wont go into or follow the path
I take it you have the complete thing:
User-agent: *
Disallow: /gallery/
depending on your servers stats you may also want to throw up an .htacces file to stop people viewing your folders (if you dont have a default file like index.html in there)
<?bhb if(broken){ echo("It wasn't me
"); } ?>
Learn HTML the ez way - EzHTML.net
Some people are like slinkies, they dont really serve any purpose but they still bring a smile to your face when you push them down the stairs ...
james.thornton posted this at 21:45 — 3rd March 2005.
They have: 42 posts
Joined: Jan 2005
For creating robots.txt files I use the free tool at:
http://www.1-hit.com/all-in-one/tool-robots.txt-generator.htm
Creates exactly what you want in your robots.txt file and has the useful capability of blocking 'nasty' robots (ones which don't index your site, just dowload your entire site to somebodies machine, wasting a large amount of your bandwidth)
James
bja888 posted this at 22:20 — 3rd March 2005.
He has: 902 posts
Joined: Feb 2004
wow! thankx for the information!!
anything else I can do to block personal content?
Busy posted this at 09:08 — 4th March 2005.
He has: 6,157 posts
Joined: May 2001
Just remember robots.txt shouldn't be used as the only means of protection, bad bots are bad bots for a reason, because they ignore the robots file.
The robots file is only a guideline for bots, the .htaccess is a command which can't be ignored
Depends what you want/need bja888
<?bhb if(broken){ echo("It wasn't me
"); } ?>
Learn HTML the ez way - EzHTML.net
Some people are like slinkies, they dont really serve any purpose but they still bring a smile to your face when you push them down the stairs ...
bja888 posted this at 10:34 — 4th March 2005.
He has: 902 posts
Joined: Feb 2004
Well... I password protected my blog site. oldbja888.com?Web-forums The mian page is a frame set. But you can get around the password by typeing in individual pages in the url. I can add ASP to all the pages and make sure they have thr framset open before they load but thats too much work. I just whan them to stop showing up when I search the net on google.
Sites hub
Flash Portfolio
OptimalSource.biz
Busy posted this at 22:27 — 4th March 2005.
He has: 6,157 posts
Joined: May 2001
if you are using cookies from the form, include a file on each page to see if the cookie is set, same for sessions, javascript is no good because you can just disable it
<?bhb if(broken){ echo("It wasn't me
"); } ?>
Learn HTML the ez way - EzHTML.net
Some people are like slinkies, they dont really serve any purpose but they still bring a smile to your face when you push them down the stairs ...