Sip Posted August 21, 2002 Report Share Posted August 21, 2002 Very interesting ... I didn't know this. For a while I had been wondering why so many go around the web looking for a file called robots.txt! THIS LINK explains it all. If you don't want any bots visiting your site, all you have to do is put this in robots.txt: code:# go awayUser-agent: *Disallow: / [ August 20, 2002, 07:08 PM: Message edited by: Sip ] Quote Link to comment Share on other sites More sharing options...
wh00t Posted August 21, 2002 Report Share Posted August 21, 2002 It's true, as a webmaster I too was once baffled by the strange robots.txt phenomenon. The "# go away" is optional. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.