Tiffany B. Brown

A web log about web development and internet culture with frequent detours into other stuff.
Fun with Scalable Vector Graphics
Reason #232 not to let someone borrow your car

How to stop (most) search engine crawlers

Most search engines adhere to the Robots Exclusion Standards when crawling a site. If you want to stop it,you’ll need to write a proper robots.txt file. How to write one, you ask? Check out the Web Robots pages.

Share this entry:
  • Digg
  • Technorati
  • del.icio.us
  • Ma.gnolia
  • Mixx
  • NewsVine
  • Reddit
  • StumbleUpon
  • TailRank
  • Furl
  • Slashdot
  • Global Grind
  • YahooMyWeb
  • Facebook
  • Google
  • Live
Comments are closed.
previous post: Fun with Scalable Vector Graphics
next post: Reason #232 not to let someone borrow your car