Tiffany B. Brown

a mish-mosh of stuff

How to stop (most) search engine crawlers

Most search engines adhere to the Robots Exclusion Standards when crawling a site. If you want to stop it,you’ll need to write a proper robots.txt file. How to write one, you ask? Check out the Web Robots pages.

Comments are closed.