The BLEXBot crawler is an automated robot that visits pages to examine and analyze the content. In this sense, it is similar to the robots used by the major search engine companies.
The BLEXBot crawler is identified by having a user-agent of the following form:
Mozilla/5.0 (compatible; BLEXBot/1.0; +https://help.seranking.com/en/blex-crawler)
The BLEXBot crawler can be identified by the user-agent above. If you are suspicious about requests being spoofed, you should first check the IP address of the request and make a reverse DNS lookup to see its domain name via appropriate tools—it should point to one of the sub-domains of *.seranking.com.
___________________________________________________________________
We care about your site's performance and will never hurt it!
BLEXbot is a very site-friendly crawler. We made it as "gentle" as possible when crawling sites: it makes only 1 request per 3 seconds, or even less frequently if another crawl delay is specified in your robots.txt file. BLEXbot respects the rules you specify in your robots.txt file.
If any problems arise, they may be due to peculiarities of your particular site or a bug on another site linking to you. Therefore, we would like to ask you, if you notice any problem with BLEXbot, please report it to crawler@seranking.com. We will quickly make unique settings for your particular site so that the crawling will never affect your site's performance.
___________________________________________________________________
Why is it crawling my site?
BLEXBot assists internet marketers in getting information on the link structure of sites and their interlinking on the web to avoid any technical and possible legal issues and improve the overall online experience. To do this, it is necessary to examine or crawl the page to collect and check all the links it has in its content.
If the BLEXBot Crawler has visited your site, this means that links have never been collected and tested on that page before or need to be refreshed. For this reason, you will not see recurring requests from the BLEXBot crawler on the same page.
The Crawler systems are engineered to be as friendly as possible, such as limiting request rates to any specific site (BLEXBot doesn't make more than one hit per 3 seconds) and automatically backing away if a site is down or slow.
___________________________________________________________________
Blocking with robots.txt
Firstly, note that BLEXBot:
- Collects only publicly available information that can be accessed by any random visitor. If you think the Crawler collects some sensitive information, please remove it from public access.
- Cannot overload your site and do any harm to it—BLEXBot is designed to be very polite, and it can make only 1 hit per 3 seconds max. Besides, you can easily slow BLEXBot (and any other robot/crawler that takes directions from the robots.txt file that should be on your site).
- Does not read, parse, collect, or store any information from your site but the links from your pages. This refers to any texts, graphical or video materials, or anything else on your pages.
With a robots.txt file, you may block the BLEXBot Crawler from parts or all of your site or slow it, as shown in the following examples:
Block specific parts of your site:
User-agent: BLEXBot
Disallow: /private/
Disallow: /messages/
Block entire site:
User-agent: BLEXBot
Disallow: /
Slow the Crawler:
User-agent: BLEXBot
Crawl-delay: 10
Attention: As soon as you make changes to your robots.txt, please give the crawler up to 10 minutes to fully stop crawling your website. This is caused by the fact that some pages might be in the processing queue already, so we cannot guarantee the crawler will be able to stop immediately. However, it should fully stop crawling after 10 minutes max.
For a general introduction to the robots.txt protocol, please see http://www.robotstxt.org. Please also see the Wikipedia article for more details and examples of robots.txt rules.
___________________________________________________________________
Contact us
All that said, we, of course, take any request to desist from crawling any site, or parts of a site, or any other feedback on the Crawler operations seriously and will act on it in a prompt and appropriate manner.
If this is the case for you, please don't hesitate to contact us at crawler@seranking.com, and we will be happy to exclude your site or investigate it immediately.