NorgBot, and how to let it in

Who we are

NORG.ai runs a web crawler called NorgBot. It reads a small number of pages from a website so we can assess how readable that site is to AI assistants, and what those assistants would find there. It is not a general-purpose search or scraping crawler, it does not index the open web, and it does not collect personal data.

If NorgBot has visited your site, it is for one of two reasons:

• Someone at your business requested an audit. This is by far the most common reason. Our free audit can only be requested by an email address at the business that owns the site.
• We were looking at your site as a business we might work with.

Verified identity

NorgBot signs every request under Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519). The public key is published at https://content-craft-api.norg.ai/.well-known/http-message-signatures-directory.

Cloudflare-fronted sites can verify NorgBot by signature and need no address-based allow rule. Each request carries Signature-Agent, Signature-Input and Signature headers; the Signature-Agent value is https://content-craft-api.norg.ai.

The user agent we send

Our origin-facing fetchers introduce themselves with a user agent containing the token NorgBot. The crawler used for audits sends:

The part after the semicolon names the job the request is doing, and it changes between requests. The stable, matchable part is always the token NorgBot.

How to allow us

Every rule below keys on the same signal: a contains match on the token NorgBot in the user agent. Pick whichever matches your setup.

Cloudflare — firewall custom rule

Dashboard → your site → Security → WAF → Custom rules → Create rule. Set the action to Skip (or Allow) and use this expression:

http.user_agent contains "NorgBot"

Place the rule above any rule that blocks bots, so it is evaluated first.

Cloudflare — the “Block AI bots” managed rule

If you have enabled Cloudflare's managed “Block AI bots” feature (Security → Bots), it can stop NorgBot before your custom rules run. Add the Skip rule above and make sure it takes precedence over the managed rule. That lets your requested audit through while leaving the managed protection in place for everything else.

WP Engine

WP Engine fronts your site with its own firewall, which is configured by address rather than by user agent — so this is the case where you should email us for the address and add it in the User Portal under your environment → Security → Allowlist / Firewall. If your plan's firewall is managed by WP Engine support, forward them the same address. If you also manage rules at the server level, the Apache rule below applies.

Apache 2.4

# Allow NorgBot through a user-agent block
<If "%{HTTP_USER_AGENT} =~ /NorgBot/">
  Require all granted
</If>

nginx

# Flag our crawler, then skip your bot block when it is set
map $http_user_agent $is_norgbot {
    default      0;
    "~*NorgBot"  1;
}

server {
    # ... inside your existing bot-blocking location:
    if ($is_norgbot) {
        break;   # skip the block for NorgBot
    }
}

Anything else

On any other firewall, WAF or CDN, the rule is the same in whatever syntax it uses: allow requests whose User-Agent header contains NorgBot, and evaluate that rule before your bot protection. If the product cannot match on user agent, ask us for the address instead.

If a user-agent rule is not enough

All NorgBot traffic originates from a single, static IP address. We do not publish it here, but we will give it to you directly if your firewall is easier to configure by address than by user agent — email solutions@norg.ai with your domain and we will send it straight back.

How NorgBot behaves

• How often we fetch. If you are seeing NorgBot for the first time, it is almost certainly because someone at your business requested a free audit of your site. That is a single pass over a small number of pages — we read them once, produce the report, and leave. We do not come back unless the audit is re-run. NORG.ai customers who have installed our edge routing are different: for them we re-read the site on a schedule (daily by default) to keep their AI-facing mirror current, skipping pages the sitemap says are unchanged. That ongoing crawl only happens on sites whose owners have set it up with us.
• We read robots.txt. NorgBot requests your robots.txt before crawling and honours Disallow directives that apply to our user agent.
• Crawl-delay. We honour the Crawl-delay directive in your robots.txt and pace our requests to match it. On a requested audit this can make the crawl take longer than it otherwise would; if you would like it to run faster, the allowlist rules above let it through at normal speed.
• What we keep. Pages we read are retained to produce and refresh the site's report, and are never used to train models.
• We do not run JavaScript on the first pass. The point of the audit is to measure what a plain, non-browser crawler receives from your server — so part of what we do is deliberately fetch your pages the unglamorous way.

Checking whether AI crawlers can reach you

An audit has to answer a question our own crawler cannot answer alone: if bot protection turns us away, does it also turn away the AI assistants your customers use? A robots.txt that welcomes them proves nothing on its own, because a firewall in front of it can refuse them anyway.

So when our crawler is refused, we request your homepage only, at most twice more, presenting the published user-agent strings of two third-party AI crawlers — OpenAI's OAI-SearchBot and Anthropic's ClaudeBot. We disclose this here rather than do it quietly.

We read the result in one direction only. If the page comes back, that is evidence AI crawlers can get in, and we say so. If it is refused, we conclude nothing and record the result as undetermined — because we send from our own address, not from OpenAI's or Anthropic's networks, so a refusal may simply mean the request did not come from where it claimed. We never report that a named crawler is blocked from your site on the strength of a probe we sent ourselves.

Stop crawling my site

Email solutions@norg.ai with the domain and we will remove it. You can also stop us yourself at any time — invert any rule above to deny the token or the address, or add a Disallow for our user agent to your robots.txt.

Questions about an audit, or about anything on this page, go to the same address.