Detectar Bots y Web Spiders de los navegadores que nos visitan

| Scripts | 11/03/2015 |
seguridad sololinux es

Actualizado el sábado, 5 mayo, 2018

Prácticamente la totalidad de los sitios web implementan su registro de visitas en una base de datos. Después de un tiempo se comienzan a acumular datos, que posteriormente se hace evidente que algunos de esos datos son basura debido a las arañas y los robots que están verificando el sitio. Este tipo de robots utilizan lineas de navegador únicas (HTTP_USER_AGENT), lo que hace que sean fáciles de identificar.

Con esta simple función, evitamos que nos contabilice como visita en nuestro registro.

function is_bot(){
 
    $bots = array(
        'Googlebot', 'Baiduspider', 'ia_archiver',
        'R6_FeedFetcher', 'NetcraftSurveyAgent', 'Sogou web spider',
        'bingbot', 'Yahoo! Slurp', 'facebookexternalhit', 'PrintfulBot',
        'msnbot', 'Twitterbot', 'UnwindFetchor',
        'urlresolver', 'Butterfly', 'TweetmemeBot' );
 
 
    foreach($bots as $b){
 
        if( stripos( $_SERVER['HTTP_USER_AGENT'], $b ) !== false ) return true;
 
    }
 
 
 
    return false;
 
}

Motores de búsqueda, arañas y rastreadores (los mas conocidos).

  • Baiduspider+(+http://www.baidu.com/search/spider.htm)
  • Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
  • Moreoverbot/5.1 (+http://w.moreover.com; webmaster@moreover.com) Mozilla/5.0
  • UnwindFetchor/1.0 (+http://www.gnip.com/)
  • Voyager/1.0
  • PostRank/2.0 (postrank.com)
  • R6_FeedFetcher(www.radian6.com/crawler)
  • R6_CommentReader(www.radian6.com/crawler)
  • radian6_default_(www.radian6.com/crawler)
  • Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
  • ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
  • Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
  • Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Mozilla/5.0 (en-us) AppleWebKit/525.13 (KHTML, like Gecko; Google Web Preview) Version/3.1 Safari/525.13
  • Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
  • Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  • Twitterbot/0.1
  • LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
  • bitlybot
  • MetaURI API/2.0 +metauri.com
  • Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
  • Mozilla/5.0 (compatible; PrintfulBot/1.0; +http://printful.com/bot.html)
  • Mozilla/5.0 (compatible; PaperLiBot/2.1)
  • Summify (Summify/1.0.1; +http://summify.com)
  • Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com)
  • PycURL/7.18.2
  • facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
  • Python-urllib/2.6
  • Python-httplib2/$Rev$
  • AppEngine-Google; (+http://code.google.com/appengine; appid: lookingglass-server)
  • Wget/1.9+cvs-stable (Red Hat modified)
  • Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)
  • Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot
  • Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
  • Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
  • Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/)
  • Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1 + FairShare-http://fairshare.cc)
  • HTTP_Request2/2.0.0beta3 (http://pear.php.net/package/http_request2) PHP/5.3.2
  • Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
  • magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
  • (TalkTalk Virus Alerts Scanning Engine)
  • Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  • Googlebot/2.1 )
  • msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
  • msnbot/2.0b (+http://search.msn.com/msnbot.htm)
  • msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
  • Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
  • Sosospider+(+http://help.soso.com/webspider.htm)
  • COMODOspider/Nutch-1.0
  • trunk.ly spider contact@trunk.ly
  • Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/)
  • Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
  • knowaboutBot 0.01
  • Showyoubot )
  • Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
  • MLBot (www.metadatalabs.com/mlbot)
  • my-robot/0.1
  • Mozilla/5.0 (compatible; woriobot support at worio dot com +http://worio.com)
  • Mozilla/5.0 (compatible; YoudaoBot/1.0; ; )
  • chilitweets.com
  • Mozilla/5.0 (TweetBeagle;
  • OctoBot/2.1 (OctoBot/2.1.0; +http://www.octofinder.com/octobot.html?2.1)
  • Mozilla/5.0 (compatible; FriendFeedBot/0.1; +Http://friendfeed.com/about/bot)
  • Mozilla/5.0 (compatible; WASALive Bot ; https://udger.com/resources/ua-list/bot-detail?bot=WASALive-Bot
  • Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)
  • urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
  • SeznamBot/3.0 (+http://fulltext.sblog.cz/)
  • Yeti/1.0 (NHN Corp.;
  • Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
  • yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_20; Asia/en) http://yacy.net/bot.html
  • Mozilla/5.0 (compatible; suggybot v0.01a,
  • ssearch_bot (sSearch Crawler; http://www.semantissimo.de)
  • Mozilla/5.0 (compatible; Linux; Socialradarbot/2.0; en-US; crawler@infegy.com)
  • wikiwix-bot-3.0
  • Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
  • Mozilla/5.0 (compatible; DotBot/1.1; , crawler@dotnetdotcom.org)
  • GarlikCrawler/1.1 (http://garlik.com/, crawler@garik.com)
  • Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
  • Mozilla/5.0 (compatible; 008/0.83; Gecko/2008032620
  • PostPost/1.0 (+http://postpo.st/crawlers)
  • Aghaven/Nutch-1.2 (www.aghaven.com)
  • SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
  • Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
  • Soup/2011-05-11Z11-51-38–soup–production-2-g251c1f9d/251c1f9d6cdff8491e0b49f4ba3288ec7f3de903 (http://soup.io/)
  • Trapit/1.1
  • Jakarta Commons-HttpClient/3.1
  • Readability/0.1
  • kame-rt (support@backtype.com)
  • Mozilla/5.0 (compatible; Topix.net;
  • Megite2.0 https://techcrunch.com/tag/megite/)
  • SkyGrid/1.0 (+http://skygrid.com/partners)
  • Netvibes (http://www.netvibes.com)
  • Zemanta Aggregator/0.7 +http://www.zemanta.com
  • Owlin.com/1.3 (http://owlin.com/)
  • Mozilla/5.0 (compatible; Twitturls; +http://twitturls.com)
  • Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) (support@tumblr.com)
  • Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0)
  • Covario-IDS/1.0 (Covario; ; support at covario dot com)

Fuente: http://www.phacks.net