php — how to detect spider/crawler with php

I am going to write a PHP script today that detects whether a search engine spider is crawling a page.With PHP, we can check the value of $_SERVER['HTTP_USER_AGENT'],a predefined variables which store the contents of the User-Agent: header from the current request, if there is one.

$pattern = ‘/(google|msnbot|MSNPTC|yahoo|Slurp|overture.com|del.icio.us|DoCoMo|DoCoMo)/i’;
if ( preg_match($pattern, $_SERVER['HTTP_USER_AGENT'] === FALSE) { // not a spider

} else { // spider

}

You can find a list of search engine spiders/crawlers fromĀ  http://www.user-agents.org/.

Comments are closed.