What does this mean?
Although I have two small sites up and running I'm new to all this server stuff. While I was checking the "Error Log" I noticed the following error 3 times all with different IP address. I know what it is it except for the robot.txt part and the File does not exist part.
File does not exist: /home/global1/public_html/robots.txt
What does it mean? Am I supposed to have a robots.txt file? I know the file doesn't exists because I built the sites. But what's going on? Do I need something for search engines?
Any insight would be appreciated.....and thanks in advance.
Greg K posted this at 03:18 — 23rd February 2005.
He has: 2,145 posts
Joined: Nov 2003
the robots.txt file is a file you can make to tell the search engines how to index your site. I haven't checked any of these out, but here is search results from Yahoo on how to set it up.
http://search.yahoo.com/bin/search?p=how+to+set+up+robots.txt
Get used to finding files in your error logs. I get tons of entries for people trying to hack their way into a windows based server.
You don't have to put in a robots.txt file, but it doesn't hurt.
-Greg
NewTechGuy posted this at 06:03 — 23rd February 2005.
He has: 57 posts
Joined: Dec 2004
Thanks......I'll read up on it.........
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.