Although I have two small sites up and running I'm new to all this server stuff. While I was checking the "Error Log" I noticed the following error 3 times all with different IP address. I know what it is it except for the robot.txt part and the File does not exist part.
File does not exist: /home/global1/public_html/robots.txt
What does it mean? Am I supposed to have a robots.txt file? I know the file doesn't exists because I built the sites. But what's going on? Do I need something for search engines?
Any insight would be appreciated.....and thanks in advance.
Focused Investments at Secular Bull
KIJHL Hockey - Revelstoke Grizzlies
Greg K posted this at 03:18—23rd February 2005.
He has: 1,601 posts
Joined: Nov 2003
the robots.txt file is a file you can make to tell the search engines how to index your site. I haven't checked any of these out, but here is search results from Yahoo on how to set it up.
http://search.yahoo.com/bin/search?p=how+to+set+up+robots.txt
Get used to finding files in your error logs. I get tons of entries for people trying to hack their way into a windows based server.
You don't have to put in a robots.txt file, but it doesn't hurt.
-Greg
[This space intentionally left blank]
Cool Geek Supplies: www.ThinkGeek.com
NewTechGuy posted this at 06:03—23rd February 2005.
He has: 57 posts
Joined: Dec 2004
Thanks......I'll read up on it.........