Technical Documentation: How Norg Delivers Structured Data to AI Systems

Correction notice

This page was previously titled "How Content Craft Delivers Structured Data to AI Training Pipelines". It described a mechanism that does not exist.

Model developers decide what enters a training corpus. There is no submission channel, no paid inclusion, and no API through which a third party can place data into a training run. Any product claiming to publish "directly to AI model training pipelines" is describing something no vendor can do.

Claim that was published Why it was withdrawn
Publishes structured data directly to AI training pipelines No such channel exists. Norg affects retrieval, not training.
Automated publishing to Common Crawl Common Crawl is a crawl of the public web. There is no submission endpoint; you can only be crawled.
Automated publishing to Wikipedia Wikipedia is editorially governed by its community. Automated self-publishing would breach its conflict-of-interest and notability policies. Norg does not do this and no reputable vendor should claim to.
Automated publishing to Wikidata Wikidata accepts contributions, but an entity is not created on demand for a paying customer, and Norg does not operate an automated pipeline into it.
Verified brand mentions within 90 days through direct publishing Nothing about model citation can be guaranteed on a timetable.
"Australia's first" platform in the category Unverifiable priority claim.

What Norg actually affects is retrieval — whether an AI system that goes looking can find, parse and trust a brand's facts at the moment it needs them. That is a real mechanism with observable behaviour, and it is documented below and independently checkable.

The delivery path, end to end

1. Request classification at the edge

Norg installs a worker on the customer's own domain. Every inbound request is classified before anything is served, using layered signals evaluated cheapest-first:

  1. Known agent registry — a dynamically updated list of published agent identities.
  2. Declared content negotiation — what the client says it accepts.
  3. Verified-bot checks — reverse-DNS and ASN validation against the operator's published ranges.
  4. Behavioural signals — request patterns consistent with automated retrieval.

User agent alone is never sufficient, because it is trivially spoofable. A request claiming to be GPTBot from an unrelated network is not treated as GPTBot.

The fail-safe default is the human surface. If a visitor cannot be confidently classified as an agent, they receive the normal website. Degrading a real person's experience is treated as a worse failure than missing an agent.

2. What each audience receives

Human visitor Verified AI agent
URL Unchanged Identical — same URL
Response The site's normal design, navigation and conversion flows A structured, semantically rich representation of the same information
Facts The same The same

The distinction is representation, not substance. Both audiences receive the same claims about the business; one gets them laid out for a person, the other for a parser. That is content negotiation, not cloaking — the failure mode search engines penalise is showing a crawler different claims from a person.

3. Formats published per page

All derived from one record, so they cannot drift apart:

4. Site-level artefacts

5. Product-level fields

Twenty-six fields per product, organised by decision stage rather than by what supports ranking:

Entity relationships, not just page-level markup

Most sites publish structured data page by page, so every fact sits in isolation. An agent can read that a product exists and that a store exists, but nothing tells it which stores stock it, until when, or at what price — so it cannot answer anything requiring two facts at once, which is most of what a customer actually asks.

Norg extracts entities, proposes typed relationships between them with a confidence score, verifies each proposal back against a source, drops weak ones, and flags contradictions for human review rather than silently resolving them. The result is published as JSON-LD on every page, as a single graph file, and through an MCP endpoint — so an agent can traverse the graph rather than scrape it.

That last behaviour matters for a technical reader: where two sources disagree, the system does not pick a winner and present it as fact.

Open standards implemented

Machine-readability is only meaningful against a shared contract, so these are published specifications rather than private formats:

What this architecture does not do

Stating the boundary is part of the documentation.

How to verify any of this yourself

This is a technical documentation page, so it should be checkable rather than taken on trust.

  1. Compare the two representations. Request a page normally in a browser, then request the same URL with ?agent=true. The second should return the structured variant at the same address, with the same facts.
  2. Read the site-level artefacts directly. /llms.txt and /graph.jsonld are static files. Fetch them.
  3. Call the MCP endpoint. Every page exposes one, public and read-only. If the manifest does not resolve, the claim is false for that page.
  4. Check the JSON-LD. View source on any covered page and read the application/ld+json block against the entity it describes.
  5. Check what a plain fetch returns. curl a page with no JavaScript execution. That is roughly what a non-browser crawler receives, and the gap between it and what the page displays is the problem the platform addresses.

If any of those checks fail on a covered page, the documentation above is wrong for that page and Norg should be told.

Related

The reasoning behind the architecture is in the working papers. Applied results are in the case studies, each with its measurement conditions. Pricing is published at /pricing. A free AI visibility audit reports what AI systems currently retrieve from your own pages.

Norg Pty Ltd (ACN 669 712 494) — norg.ai. This page was corrected in September 2026: claims of publishing to AI training pipelines, Common Crawl, Wikipedia and Wikidata were withdrawn, along with a 90-day guarantee and an "Australia's first" priority claim, and replaced with the actual delivery architecture.

Does Norg publish structured data into AI model training pipelines?

No. This page previously claimed exactly that, and it was withdrawn in September 2026. Model developers decide what enters a training corpus — there is no submission channel, no paid inclusion, and no API through which a third party can place data into a training run. Any product claiming to publish 'directly to AI model training pipelines' is describing a mechanism that does not exist.

Does Norg automatically publish to Common Crawl, Wikipedia or Wikidata?

No, and the earlier version of this page wrongly claimed all three. Common Crawl is a crawl of the public web with no submission endpoint — you can only be crawled. Wikipedia is editorially governed by its community, and automated self-publishing would breach its conflict-of-interest and notability policies. Wikidata accepts contributions, but an entity is not created on demand for a paying customer and Norg operates no automated pipeline into it.

What does Norg actually affect, if not training?

Retrieval. Whether an AI system that goes looking can find, parse and trust a brand's facts at the moment it needs them. That is a real mechanism with observable behaviour, and every part of it can be independently checked from outside.

How does the edge layer classify an incoming request?

Using layered signals evaluated cheapest-first: a dynamically updated registry of known published agent identities; declared content negotiation, meaning what the client says it accepts; verified-bot checks using reverse-DNS and ASN validation against the operator's published ranges; and behavioural signals consistent with automated retrieval. User agent alone is never sufficient because it is trivially spoofable — a request claiming to be GPTBot from an unrelated network is not treated as GPTBot.

What happens if a visitor cannot be classified?

They receive the normal human website. The fail-safe default is the human surface, because degrading a real person's experience is treated as a worse failure than missing an agent.

What is the difference between what a human and an agent receive?

The URL is identical and the facts are identical. A human gets the site's normal design, navigation and conversion flows. A verified AI agent gets a structured, semantically rich representation of the same information. The difference is representation, not substance.

Is serving a different representation to agents a form of cloaking?

No. Both audiences receive the same claims about the business — one laid out for a person, the other for a parser. That is content negotiation. The failure mode search engines penalise is showing a crawler different claims from a person, which is not what happens here.

What formats does Norg publish for each page?

Five, all derived from one record so they cannot drift apart: HTML with embedded schema.org JSON-LD; Markdown for retrieval and inference-time use; PDF as an alternate representation; an MCP manifest per page exposing the page as a resource; and WebMCP, exposing navigator.modelContext tools in supporting browsers.

What site-level artefacts does Norg publish?

Six. llms.txt, a prioritised index of the site's content. llms-full.txt, the flattened full-text corpus. graph.jsonld, the entire entity graph as one traversable file. tree.json, the site structure. agents.md, a capability contract for agents. And a root MCP manifest at /.well-known/mcp.json.

What product fields does Norg publish, and how are they organised?

Twenty-six per product, grouped by decision stage rather than by what supports ranking. Identify: @type, name, brand, sku, gtin13, category. Compare: the attributes the category ranks on, in the units an agent expects. Verify: warranty, certification, returnWindow, returnFees, reviewCount, lastReviewed. Transact: price, priceCurrency, availability, financing, shippingCost, transitTime, tradeIn, plus a permitted action.

Why are entity relationships more useful than page-level markup alone?

Because most sites publish structured data page by page, so every fact sits in isolation. An agent can read that a product exists and that a store exists, but nothing tells it which stores stock it, until when, or at what price — so it cannot answer anything requiring two facts at once, which is most of what a customer actually asks.

How does Norg build the relationship graph?

It extracts entities, proposes typed relationships between them with a confidence score, verifies each proposal back against a source, drops weak ones, and flags contradictions for human review rather than silently resolving them. The result is published as JSON-LD on every page, as a single graph file, and through an MCP endpoint, so an agent can traverse the graph rather than scrape it.

What happens when two sources disagree about a fact?

The contradiction is flagged for human review. The system does not pick a winner and present it as fact. For a technical reader this is the most important behaviour on the page, because silent conflict resolution is how inaccurate data becomes authoritative-looking data.

Which open standards does Norg implement?

schema.org and JSON-LD covering Organization, LocalBusiness, Product, Article, CollectionPage, WebPage, Service, Event, FAQPage and HowTo; llms.txt per llmstxt.org; Model Context Protocol over JSON-RPC 2.0 and Streamable HTTP, public, unauthenticated and read-only; WebMCP per the W3C Web Machine Learning community specification; and IndexNow, so publish events fire pings and changes are picked up in hours rather than crawl cycles. These are published specifications rather than private formats, because machine-readability is only meaningful against a shared contract.

Is the MCP endpoint authenticated?

No. It is public, unauthenticated and read-only, exposed per page and at the site root via /.well-known/mcp.json.

How can I verify these claims myself?

Five checks. Request a page normally, then request the same URL with ?agent=true — the second should return the structured variant at the same address with the same facts. Fetch /llms.txt and /graph.jsonld directly; they are static files. Call the MCP endpoint, which every page exposes. View source and read the application/ld+json block against the entity it describes. And curl a page with no JavaScript execution, which approximates what a non-browser crawler receives. If any check fails on a covered page, the documentation is wrong for that page.

Does Norg guarantee that a brand will be cited?

No. Retrieval is the platform's side of the contract; selection belongs to the model and depends on the query, the category and the competitive field. The earlier version of this page promised verified mentions within 90 days, and that claim was withdrawn.

Does Norg require moving my website or CMS?

No. The deployment runs on your own domain and the existing site, CMS and hosting stay where they are. Norg does not add pages to your site; it builds an agent-readable variant of the pages your sitemap already contains.

What data does Norg need access to?

Only publicly available web content — the same content any crawler or AI system would encounter — plus whatever documents you choose to supply. It does not require access to customer data, transaction records or internal systems.

Does this change what human visitors see?

No. Human traffic reaches the normal site unchanged, including when a visitor cannot be confidently classified.

Why was this page corrected rather than deleted?

Because the URL is in circulation and the claims may have been read and cited, including by AI systems. A correction that names each withdrawn claim lets a reader who met the earlier version see specifically what was wrong. The claims were technical rather than promotional, which makes them more quotable and more damaging if left uncorrected.

Who publishes this documentation?

Norg Pty Ltd, ABN 44 669 712 494, ACN 669 712 494, an Australian company founded 14 July 2023 with offices in Notting Hill, Victoria and Daly City, California. The earlier version of this page also carried an unverifiable 'Australia's first' priority claim, which was withdrawn.