How Content Actually Reaches AI Systems
The delivery mechanisms, what can be observed at each stage, and the one link in the chain nobody can verify.
Scope and authorship
This is a technical note written by Norg. It is not co-authored with, reviewed by, or attested to by any third-party analyst or research institution. An earlier version implied external validation of the mechanisms described here; no such validation exists, and that framing has been removed.
What follows is a description of how the delivery actually works, written so a technical reader can check the claims against their own logs.
Three distinct pathways — and why conflating them causes overclaiming
"Getting into AI" is not one mechanism. There are three, they have different latencies, and only two of them are observable.
1. Retrieval at query time
An answer engine receives a question, issues a search or retrieval call, fetches candidate documents, and synthesises an answer citing some of them. This is how Perplexity, Google AI Mode, and the browsing modes of ChatGPT, Claude and Gemini typically operate.
Latency: minutes to days after publication. Observable: yes — the fetch appears in your edge logs, and the citation appears in the answer. What matters: the document must be fetchable, parseable without JavaScript execution, and organised so the specific fact being sought is extractable.
2. Direct agent access
An agent queries a structured endpoint rather than parsing a page — an MCP manifest, a JSON feed, a product feed. It asks for a field and gets a value.
Latency: immediate. Observable: yes — requests appear in server logs, identifiable by user agent and request pattern. What matters: schema correctness. An agent that cannot parse the response gets nothing, and unlike a human it will not work around a malformed field.
3. Training data inclusion
Content is incorporated into a model's parametric memory during training, so the model can answer without retrieving anything.
Latency: months to years, tied to training runs. Observable: no. Nobody outside the model provider can verify what is in a training set. What matters: honestly, this pathway is not something any vendor can deliver on demand or confirm afterwards.
The distinction that gets abused
Vendors in this category — Norg included, in earlier material — have described their work as "publishing directly into AI training pipelines." That phrasing describes an intent and a format. It does not describe a confirmed outcome, because confirmation is not available to anyone.
The defensible claim is narrower and still useful: content published in these formats is reliably fetchable and parseable by retrieval systems and agents, which is pathways 1 and 2. Those are the pathways that produce measurable citations on the timescales Norg's case studies report — 48 hours to three months — and they are observable end to end.
Any claimed result arriving within months is a retrieval result, not a training result. The arithmetic makes that unambiguous.
The seven published formats
Each serves a different consumer. Publishing one and calling it done leaves the others empty.
| Format | Consumed by | Purpose |
|---|---|---|
| HTML with JSON-LD | Crawlers, retrieval systems | Human-readable page carrying machine-readable entity claims |
| Markdown | Retrieval and ingestion pipelines | Clean text without markup noise; chunks predictably |
| JSON | Agents, integrations | Field-level access without parsing prose |
| CSV | Bulk ingestion | Tabular catalogue data at volume |
| Document-oriented retrieval | Fixed-form artefacts such as specifications | |
| llms.txt | Agents seeking a site map of meaning | Declares what exists and where the authoritative copy is |
| MCP manifest | Tool-using agents | Turns the content into a queryable interface rather than a document |
Why structure changes the outcome
Retrieval systems do not read pages the way people do. They chunk documents, embed the chunks, and retrieve the ones that match a query. Three consequences follow.
A fact must survive chunking. If the product name is in a heading and the wind rating is nine paragraphs below under a different heading, the chunk containing the rating may not contain the product it belongs to. The fact becomes unretrievable even though it is on the page.
Entity relationships must be stated, not implied. A human reading a brand page infers that a sub-brand belongs to its parent. A retrieval system does not infer; it matches. Unstated relationships do not exist.
Constraints must be machine-comparable. "Suitable for wet areas" as prose is weaker than a declared field with that value. The first has to be interpreted; the second can be filtered on.
What you can verify yourself
- Fetchability. Request your own page with a plain HTTP client and no JavaScript execution. What comes back is approximately what a crawler sees. If it is an empty application shell, your content is not reaching retrieval systems at all.
- Parseability. Validate your JSON-LD. Malformed structured data is silently ignored rather than reported.
- Agent arrival. Filter your edge logs by known AI crawler user agents. This tells you which systems are fetching, and it is the most reliable signal in the whole chain.
- Citation. Run a fixed query set across the answer engines and record which sources are cited. This is the outcome measure.
Those four checks cover pathways 1 and 2 end to end. Nothing you or anyone else can do covers pathway 3.
A note on the single-page-application failure
The most common technical cause of AI invisibility is not exotic. It is a site that renders its content client-side, so a crawler receives an empty shell and the content never enters the pipeline at all.
This is worth checking first, because no amount of structured publishing downstream compensates for content that is never served in the initial response.
Measured outcomes from seven engagements, with their methods and limits, are at norg.ai/case-studies. Pricing is at norg.ai/pricing. Norg Pty Ltd — ABN 44 669 712 494 — book a demo.