SERankingPCFBot fetches publicly available web pages to build search and answer-engine visibility reports for SE Ranking customers. PCF stands for Page Content Fetch.
The crawler is identified by the following user-agent:
Mozilla/5.0 (compatible; SERankingPCFBot/1.0; +https://seranking.com/pcf-crawler)
Verification
The user-agent above can be spoofed. To confirm that a request is genuinely from SERankingPCFBot, use one of the following methods.
- Reverse DNS. A reverse DNS lookup on the requesting IP returns a subdomain of
[SUFFIX]. Run a forward lookup on that hostname and confirm it resolves back to the original IP address. - IP address list. All requests originate from the addresses published at
[ENDPOINT]. This list contains the addresses used by SERankingPCFBot only, and is always up to date. Plain text format is available at[ENDPOINT]?txt=1. - Signed requests. SERankingPCFBot signs its requests using HTTP Message Signatures. Our public keys are published at
[KEY_DIRECTORY].
Crawling behaviour
SERankingPCFBot always respects the rules specified in your robots.txt file. There is no setting that allows this crawler to ignore robots.txt.
If no crawl delay is specified in your robots.txt, or the file is unavailable, the default crawl delay is [N] seconds.
SERankingPCFBot requests any individual page no more often than [FREQ].
The crawler collects only publicly available information that any visitor can access, and does not use the collected content to train machine learning models.
If your server returns 4xx or 5xx responses, the crawler slows down automatically.
Managing SERankingPCFBot
Please ensure your robots.txt returns an HTTP 200 status code. If it does not, the crawler will treat the file as non-existent.
Slow the crawler down:
User-agent: SERankingPCFBot Crawl-delay: 10
Block part of your site:
User-agent: SERankingPCFBot Disallow: /private/
Block your entire site:
User-agent: SERankingPCFBot Disallow: /
After you change your robots.txt, please allow up to one day for the crawler to fully stop. Some pages may already be queued for processing.
Contact us
If you notice any problem with SERankingPCFBot, write to help@seranking.com and we will investigate immediately.