# Google adsbot ignores robots.txt unless specifically named! User-Agent: AdsBot-Google Disallow: / User-agent: Python/3.5 aiohttp/1.0.5 Disallow: / User-agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com) Disallow: / User-agent: Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot) Disallow: / User-agent: Mozilla/5.0 Mozilla/5.0 (compatible; ArhpostBot/0.1; https://arhpost.ru/; ru-RU) Disallow: / User-agent: Mozilla/5.0 BoogleBot 2.0 Disallow: / User-agent: Mozilla/5.0 (compatible; memoryBot/1.24.61 +http://internetmemory.org/en/) Disallow: / User-agent: Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html) Disallow: / User-agent: uclassify.com/1.0 Disallow: / User-agent: SuperPagesUrlVerifyBot/1.0 Disallow: / User-agent: Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html Disallow: / User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) Disallow: / User-agent: vebidoobot Disallow: / User-agent: Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1 Disallow: / User-agent: http:/best-muzon.org/popsa/1202-t-killah-i-nastya-kochatkova-nad-zemley.html Disallow: / User-agent: http:/best-muzon.org/house_dance/238-avicii-wake-me-up-sanana-bootleg.html Disallow: / User-agent: redback/v0-570-g26f8c96 Disallow: / User-agent: http:/best-muzon.org/popsa/1600-maks-lidov-otrazheniya.html Disallow: / User-agent: http:/best-muzon.org/popsa/2144-louisa-johnson-so-good.html Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE3.00; Windows 2003) Disallow: / User-agent: Mozilla/4.79 [en] (Windows NT 5.0; U) Disallow: / User-agent: BoogleBot2 Disallow: / User-agent: Mozilla/5.0 (compatible; Optimizer) Disallow: / User-agent: Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot) Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html Disallow: / User-agent: Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot) Disallow: / User-agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36 (BingLocalSearch) Disallow: / User-agent: NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) Disallow: / User-agent: Node/simplecrawler 0.7.0 (git+https://github.com/cgiffard/node-simplecrawler.git) Disallow: / User-agent: Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com) Disallow: / User-agent: Mozilla/5.0 (compatible; RukiCrawler/1.0; +http://ruki.rezko.net) Disallow: / User-agent: Mozilla/0.91 Beta (Windows) Disallow: / User-agent: Scrapy/1.2.1 (+http://scrapy.org) Disallow: / User-agent: Lynx/2.8.5rel.1 libwww-FM/2.15FC SSL-MM/1.4.1c OpenSSL/0.9.7e-dev Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Aboundex/0.3 Disallow: / User-agent: python-requests/2.11.1 Disallow: / User-agent: python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-100-generic Disallow: / User-agent: Aboundexbot Disallow: / User-agent: curl/7.47.0 Disallow: / User-agent: panscient.com Disallow: / User-agent: r/curl/jeroen Disallow: / User-agent: curl/7.38.0 Disallow: / User-agent: Wappalyzer; +https://github.com/AliasIO/Wappalyzer Disallow: / User-agent: PycURL Disallow: / User-agent: scrapy-redis (+https://github.com/rolando/scrapy-redis) Disallow: / User-agent: VidibleScraper/1.0 Disallow: / User-agent: integrity/5 Disallow: / User-agent: PHPCrawl Disallow: / User-agent: CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html) Disallow: / User-agent: WBSearchBot/1.1 Disallow: / User-agent: DeuSu/5.0.2 Disallow: / User-agent: msnbot Disallow: / User-agent: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) Disallow: / User-agent: DomainAppender /1.0 Disallow: / User-agent: LinkedInBot/1.0 Disallow: / User-agent: LinkpadBot/1.07 Disallow: / User-agent: TeeRaidBot Disallow: / User-agent: Powermarks/3.5 Disallow: / User-agent: GrapeshotCrawler/2.0 Disallow: / User-agent: coccoc/1.0 Disallow: / User-agent: CheckMarkNetwork/1.0 Disallow: / User-agent: Dataprovider/6.101 Disallow: / User-agent: Mozilla/5.0 (compatible; Dataprovider/6.101; +https://www.dataprovider.com/) Disallow: / User-agent: Python-urllib/2.7 Disallow: / User-agent: DuckDuckGo-Favicons-Bot/1.0 Disallow: / User-agent: CCBot/2.0 Disallow: / User-agent: Metaspinner/1.0 Disallow: / User-agent: Dalvik/2.1.0 Disallow: / User-agent: MojeekBot/0.6 Disallow: / User-agent: oBot/2.3.1 Disallow: / User-agent: KomodiaBot/1.0 Disallow: / User-agent: wonderbot/JS 1.0 Disallow: / User-agent: Ruby Disallow: / User-agent: woobot/1.1 Disallow: / User-agent: woobot/2.0 Disallow: / User-agent: DowntimeDetector/1.0 Disallow: / User-agent: Uptimebot/1.0 Disallow: / User-agent: Dispatch/0.11.3 Disallow: / User-agent: eventBot Disallow: / User-agent: InternetSeer.com Disallow: / User-agent: SynHttpClient-Built/5843 Disallow: / User-agent: Go-http-client/1.1 Disallow: / User-agent: SemrushBot/1.1~bl Disallow: / User-agent: XoviBot Disallow: / User-agent: XoviBot/2.0 Disallow: / User-agent: SeznamBot Disallow: / User-agent: DuckDuckBot Disallow: / User-agent: Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98) Disallow: / User-agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) Disallow: / User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) Disallow: / User-agent: ia_archiver Disallow: / User-agent: spbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: WBSearchBot Disallow: / User-agent: rogerbot Disallow:/ User-Agent: AhrefsBot Disallow: / User-agent: IstellaBot Disallow: / User-agent: SiteExplorer Disallow: / User-agent: SiteLockSpider Disallow: / User-agent: SiteLockSpider [en] (WinNT; I ;Nav) Disallow: / User-agent: SiteLock Disallow: / User-agent: facebot Disallow: / User-agent: facebookexternalhit/1.1 Disallow: / User-agent: facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php) Disallow: / User-agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) Disallow: / User-agent: Mail.Ru Disallow: / User-agent: Mail.RU_Bot/2.0 Disallow: / User-agent: wotbox Disallow: / User-agent: Yahoo! Slurp Disallow: / User-agent: aiHitBot Disallow: / User-agent: aiHitBot/2.9 Disallow: / User-agent: 360Spider Disallow: / User-agent: Arachnida Disallow: / User-agent: Associative Spider Disallow: / User-agent: Baiduspider disallow: / User-agent: MJ12bot Disallow: / User-agent: WOW64 Disallow: / User-agent: MRA 4.6 Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13 Disallow: / User-agent: TwengaBot-Discover Disallow: / User-agent: TwengaBot/2.0 Disallow: / User-agent: TwengaBot Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler) Disallow: / User-agent: Yandex Disallow: / User-agent: Yandexbot Disallow: / User-agent: BLEXBot Disallow: / User-agent: iOpus-I-M Disallow: / User-Agent: sitecheck.internetseer.com Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: dotbot Disallow: / User-agent: FreeFind Disallow: / User-agent: BecomeBot Disallow: / User-agent: Nutch Disallow: / User-agent: Jetbot/1.0 Disallow: / User-agent: Jetbot Disallow: / User-agent: WebVac Disallow: / User-agent: Stanford Disallow: / User-agent: naver Disallow: / User-agent: dumbot Disallow: / User-agent: Hatena Antenna Disallow: / User-agent: grub-client Disallow: / User-agent: grub Disallow: / User-agent: WebZip Disallow: / User-agent: larbin Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: Copernic Disallow: / User-agent: psbot Disallow: / User-agent: Python-urllib Disallow: / User-agent: NetMechanic Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: CherryPicker Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: WebBandit Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: Crescent Disallow: / User-agent: SiteSnagger Disallow: / User-agent: ProWebWalker Disallow: / User-agent: CheeseBot Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: Mozilla Disallow: / User-agent: mozilla Disallow: / User-agent: mozilla/3 Disallow: / User-agent: mozilla/4 Disallow: / User-agent: mozilla/5 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) Disallow: / User-agent: Alexibot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Stanford Comp Sci Disallow: / User-agent: MIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: Website Quester Disallow: / User-agent: moget/2.1 Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: WebStripper Disallow: / User-agent: WebSauger Disallow: / User-agent: WebCopier Disallow: / User-agent: NetAnts Disallow: / User-agent: Mister PiX Disallow: / User-agent: WebAuto Disallow: / User-agent: TheNomad Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: RMA Disallow: / User-agent: libWeb/clsHTTP Disallow: / User-agent: asterias Disallow: / User-agent: httplib Disallow: / User-agent: turingos Disallow: / User-agent: spanner Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Microsoft URL Control - 5.01.4511 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: Foobot Disallow: / User-agent: SpankBot Disallow: / User-agent: BotALot Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: lwp-trivial Disallow: / User-agent: http://www.WebmasterWorld.com bot Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Microsoft URL Control - 6.00.8169 Disallow: / User-agent: URLy Warning Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget Disallow: / User-agent: LinkWalker Disallow: / User-agent: cosmos Disallow: / User-agent: moget Disallow: / User-agent: hloader Disallow: / User-agent: humanlinks Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Mata Hari Disallow: / User-agent: LexiBot Disallow: / User-agent: Web Image Collector Disallow: / User-agent: The Intraformant Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: True_Robot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: http://www.SearchEngineWorld.com bot Disallow: / User-agent: http://www.WebmasterWorld.com bot Disallow: / User-agent: JennyBot Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: WebEnhancer Disallow: / User-agent: suzuran Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCI Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: Openfind Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Xenu's Disallow: / User-agent: Zeus Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RepoMonkey Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: Openbot Disallow: / User-agent: URL Control Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Webster Pro Disallow: / User-agent: EroCrawler Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: FairAd Client Disallow: / User-agent: Gaisbot Disallow: / User-agent: Aqua_Products Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: MSIECrawler Disallow: / User-agent: PerMan Disallow: / User-agent: searchpreview Disallow: / User-agent: sootle Disallow: / User-agent: es Disallow: / User-agent: Enterprise_Search/1.0 Disallow: / User-agent: Enterprise_Search Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 Disallow: / User-agent: bingbot Disallow: / User-agent: bing Disallow: / User-agent: googlebot User-agent: google Disallow: / User-agent: Baiduspider User-agent: Baiduspider-video User-agent: Baiduspider-image Disallow: / User-agent: * Disallow: /