Website Audit FAQ

The most common reason for the Website Audit failing to start is our bot getting blocked by your website’s firewall.

To ensure the proper launch and scan please make sure that our bot isn’t blocked by the firewall by whitelisting our crawler's IPs and user-agent(SEBot-WA).

Please note, that our bot operates from German IPs, and security rules that block access from Germany may override the whitelist.

The default limit for the number of scanned pages is set to 1000, you may change it in the tool’s settings.

The default speed is set to 500 requests per second, but you may limit it in the tool’s settings.

Some servers, CDNs, and security services (for example Cloudflare) may block or throttle the audit crawler, causing it to slow down, stall, or fail. The fix depends on which error you're seeing.

Start here for any of these errors. Whitelist SE Ranking’s crawler IPs and the SEBot-WA user-agent, as described in the Verifying SE Ranking Crawler FAQ. If you use Cloudflare, also follow the “How to whitelist SE Rankings IPs in Cloudflare?” section below and disable Bot Fight Mode. SEBot-WA is the default user-agent — if you have selected a different one in the Website Audit settings, whitelist that one instead.

403 (Forbidden) — the bot is being blocked outright. Whitelist our IPs and user-agent as above. If the error persists, check for additional blocking rules that can override a whitelist — for example, region-specific blocking (our bot operates from German IPs) or rules that block bots that don’t render JavaScript.

403 that survives whitelisting — try JavaScript rendering. If whitelisting the IPs and the user-agent doesn't help, the block may come from a rule that challenges visitors with JavaScript. A manual test can pass while the audit still fails: a request such as curl -I -A "SEBot-WA" https://yourdomain.com/ can return HTTP 200 because it isn't challenged, while the crawl is. Enabling JavaScript rendering makes the audit fetch your pages through a browser session that executes JavaScript instead of a plain HTTP request, which can get past such a challenge. Open Settings in the Website Audit, go to the Parser settings tab, turn on the JavaScript rendering option, and restart the audit. This is worth trying, but it is not guaranteed to resolve every 403 — if the block comes from something other than a JavaScript challenge, the audit will still fail.

429 (Too Many Requests) — the server is rate-limiting the crawler. Lower the crawl speed by reducing the maximum number of requests in the Website Audit settings.

503 (Service Unavailable) or timeout — a server-side error. This can have several causes. Lowering the crawl speed (same settings as above) sometimes helps, because a fast crawl can put more load on the server than it can handle. If that doesn’t resolve it, your hosting provider will need to investigate the underlying cause.

If the audit still fails after trying the relevant steps above, please contact our support team and include the error message shown in the audit.

An outbound (external) link is reported as broken when the third-party domain returns a 4XX status to our crawler. External links are verified with a lightweight HEAD request rather than a full page load, and many large sites block, challenge, or simply refuse such automated requests. That is why a page which opens normally in your browser can still answer 403 or 404 for the audit. The link is not broken for your visitors, but it is still counted in the report and can lower your website health score.

To stop specific third-party domains from being reported this way, exclude them from external-link checking in the Website Audit settings: open Settings, go to the Rules for scanning pages tab, and add the domains or URLs to the block for ignoring external links. Excluded addresses are then skipped when external links are queued for checking, and the affected issues drop out of the report on the next audit. This rule applies to outbound links only — it does not change which of your own pages are crawled, which is controlled by the separate allow, disallow and hide path rules on the same tab.

By default, the Website Audit uses the Site pages source: it starts from your homepage and follows every internal link it finds, rather than relying only on your XML sitemap. As a result, it can reach and analyze pages that are not listed in your sitemap. A noindex directive only asks search engines not to index a page — it does not stop a crawler from fetching it. That is why noindex and non-sitemap pages are still crawled and can be flagged for issues such as “HTML file too large”.

To narrow what gets crawled, open the tool’s settings and set Source of pages to XML sitemap (or upload My list of pages) so only those URLs are scanned, add disallow paths to exclude specific directories, and keep the Ignore Noindex option disabled so noindex pages are skipped.

This guide is especially important if you actively use SE Ranking’s Website Audit and On-Page SEO Checker.

The Cloudflare Firewall provides protection from vulnerabilities such as malicious incoming traffic. Sometimes, its automatic rules block our crawlers and prevents SE Ranking’s bots from crawling your website.

But you can easily remove this restriction because Cloudflare provides users with great flexibility. Just follow these steps:

1. Log in to the Cloudflare dashboard, and select your account and domain.
2. Go to the Security section > click WAF > choose the Tools subsection
3. Under IP Access Rules, first enter one of SE Ranking’s IPs. The rule must be set for each crawler's IP separately.

Select a Skip action. Then for Zone, select whether the rule applies to the current website only or to all websites in the account.

Finally, do the same steps with other IPs.

That’s it! If you want to know more, go to the official Cloudflare guide.

The website audit starts automatically when you create a new project. To restart the audit, click the appropriate button in the Website Audit module. Also, you can set up a schedule for automatic rechecks – once per week or month in the Settings. Every run is a full new crawl — results from a previous audit are not reused — so a fix you have published will be picked up as soon as the next audit finishes.

Since your website is checked against dozens of parameters, we can’t predict exactly how long it will take the system to complete the audit. It depends on the number of pages and images on the website, whether there is a crawl delay in the robots.txt file, as well as the internal queue in the system. You will be able to monitor the crawling process in the Website Audit module.

You will get a notification via email once the audit report has been generated.

The Website Audit scans all pages of a website and provides an analysis along with suggestions on how to fix identified problems. An example of the report can be found here.

The On-Page SEO Checker audits a certain URL and evaluates how well it is optimized for a certain query. It detects mistakes that negatively affect the given page’s search engine ranking. An example of the report can be found here.

In order to confirm that, you must know how to verify SE Ranking Crawler. Please check our quick guide, which explains how to do that. By default, the Website Audit crawler sends the SEBot-WA user-agent, so that is the value to look for in your server logs and to allow in your firewall rules. If you have chosen a different user-agent in the tool's settings, look for that one instead.

A stand-alone (non-project) audit can be shared by downloading its PDF report or by sending the report by email (including to a sub-account’s email address), as described in How to use Website Audit. Assigning or sharing a stand-alone audit directly with a sub-account inside the tool is not currently supported.