Identifying Search engines > Support

Unanswered Active Topics

Forums Search

Forums > Other Products > Site Analyzer

Subject: Identifying Search engines

Prev Next

You are not authorized to post a reply.

Author

Messages

Georg Jansen
Posts:5
Bronze Member

12/21/2005 2:06 PM
Seems like not all search engines is recognized in the database function dbo.BizModules_IsBot I changed the function (se bellow): ALTER function dbo.BizModules_IsBot ( @Agent as varchar(50) ) returns bit as begin if CharIndex('bot',@Agent)>0 return 1 if CharIndex('SpiderMan',@Agent)>0 return 1 if CharIndex('Search',@Agent)>0 return 1 --- New tests inserted if @Agent='LinkWalker' return 1 if @Agent like '%Slurp%' return 1 if @agent like '%Ask Jeeves%' return 1 return 0 end

Best Regards Georg Jansen www.l4ndash.com - Log4Net Dashboard www.Faktnet.com

Pengtsen R
Posts:5191

12/21/2005 5:29 PM
Thanks for sharing your discovery. I'll integrate it into the next release.

Pengtsen R http://www.bizmodules.net

Georg Jansen
Posts:5
Bronze Member

12/21/2005 6:30 PM
Just to give you some more background on this, what I did was to collect the names of all the UserAgents that retrieved the Robots.txt file from my site, I enclose the list – in case I have missed someone in my modified version of the function. g2Crawler+(nobody@airmail.net) Baiduspider+(+http://www.baidu.com/search/spider.htm) EuripBot/0.2+(+http://www.eurip.com)+GetRobots Everest-Vulcan+Inc./0.1+(R&D+project;+http://everest.vulcan.com/crawlerhelp) exactseek-pagereaper-2.63+(crawler@exactseek.com) Googlebot-Image/1.0 Googlebot/2.1+(+http://www.google.com/bot.html) http://www.almaden.ibm.com/cs/crawler ia_archiver larbin_2.6.3+(larbin2.6.3@unspecified.mail) libwww-perl/5.800 LinkWalker Mediapartners-Google/2.1 MJ12bot/v1.0.6+(http://majestic12.co.uk/bot.php?+) Mozilla/2.0+(compatible;+Ask+Jeeves/Teoma) Mozilla/2.0+(compatible;+Ask+Jeeves/Teoma;++http://sp.ask.com/docs/about/tech_crawling.html) Mozilla/4.0+(compatible:+FDSE+robot) Mozilla/4.0+(compatible:+Shoula+robot) Mozilla/4.0+(compatible;+MSIE+5.0;+Windows+NT;+Girafabot;+girafabot+at+girafa+dot+com;+http://www.girafa.com) Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0;+obot) Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0;+.NET+CLR+1.0.3705;+.NET+CLR+1.1.4322;+InfoPath.1;+.NET+CLR+2.0.50727;+MSIECrawler) Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0;+.NET+CLR+1.0.3705;+MSIECrawler) Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.1.4322;+MSIECrawler) Mozilla/4.0+(compatible;+T-H-U-N-D-E-R-S-T-O-N-E) Mozilla/5.0+(compatible;+BecomeBot/2.3;+MSIE+6.0+compatible;++http://www.become.com/site_owners.html) Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp) msnbot/0.9+(+http://search.msn.com/msnbot.htm) msnbot/1.0+(+http://search.msn.com/msnbot.htm) NG/2.0 psbot/0.1+(+http://www.picsearch.com/bot.html) RufusBot+(Rufus+Web+Miner;+http://64.124.122.252/feedback.html) Scrubby/2.1+(http://www.scrubtheweb.com/abs/meta-check.html) Search+Engine+World+Robots.txt+Validator+at+http://www.searchengineworld.com/cgi-bin/robotcheck.cgi SurveyBot/2.3+(Whois+Source) Websquash.com+(Add+url+robot) Yahoo-MMCrawler/3.x+(mms+dash+mmcrawler+dash+support+at+yahoo+dash+inc+dot+com)

Best Regards Georg Jansen www.l4ndash.com - Log4Net Dashboard www.Faktnet.com

Pengtsen R
Posts:5191

12/23/2005 10:07 AM
Thanks for your post. I really appreciate it.

Pengtsen R http://www.bizmodules.net

You are not authorized to post a reply.

Forums > Other Products > Site Analyzer > Identifying Search engines