Why Peek-Then-Pay Enables a Better Architecture for Publishers and LLMs
Gary Newcomb
CTO & Co-Founder, FetchRight
The rapid rise of AI agents and large language models has forced publishers and platform operators into a difficult position.
On one side, publishers want their content to be understood by AI systems. They want their brand, expertise, and editorial voice represented accurately in AI answers and recommendations.
On the other side, they want control over how their intellectual property is accessed, transformed, and monetized.
Meanwhile, AI platforms face the opposite challenge: they need high-quality knowledge sources that are structured, reliable, and legally accessible. But today they are often forced to scrape raw HTML and reverse engineer meaning from it.
The current ecosystem is messy because three fundamentally different responsibilities are being blended together:
These concerns are often implemented in a single system: a CDN feature, a crawler policy, or an ad-hoc licensing API.
But when these responsibilities are combined, both publishers and AI systems suffer.
The FetchRight / Peek-Then-Pay model proposes something different:
A clean separation of concerns where each responsibility is handled in the layer where it belongs.
This architecture benefits both publishers and AI platforms.
To understand why this matters, it's helpful to look at the three core functions separately.
Licensing answers questions like:
Licensing should be centralized.
Why?
Because licensing is fundamentally a business relationship, not a runtime infrastructure concern.
A publisher may negotiate agreements with multiple AI companies, each with different policies, pricing models, or permitted uses.
Centralized licensing systems allow:
In the FetchRight architecture, licensing is handled by a central license authority, but runtime systems do not depend on it for every request.
That's where the second concern comes in.
Traditional API systems rely on centralized accounting; every request must contact a license server.
At internet scale, this creates problems:
FetchRight uses a different model:
Assertion-only licenses with local bookkeeping.
This means:
This approach offers several advantages:
Local metering systems (whether Redis, Durable Objects, or other storage) track usage in real time.
Licensing remains centralized. Accounting remains distributed.
Each system does the job it is best suited for.
The third concern is often overlooked but may be the most important.
AI systems rarely consume raw HTML.
They transform content into:
Today, these transformations are typically performed by:
But this introduces a critical problem.
The entity transforming the content is often not the publisher.
Which means:
This is why FetchRight treats content transformation as a publisher-controlled responsibility.
The publisher decides:
This is where modern AI tooling becomes incredibly powerful.
Publishers already understand their content better than anyone else.
They know:
Using modern AI models internally, publishers can generate representations that preserve this knowledge.
For example, a publisher could generate embeddings using systems like Gemini Embeddings 2.
These embeddings might incorporate:
Instead of exposing raw HTML to be scraped, the publisher exposes a canonical semantic representation of their content.
This benefits everyone.
From the perspective of an AI platform, publisher-generated transformations offer several advantages.
Generic crawlers must infer meaning from messy web pages.
Publishers can provide clean semantic representations built from the original source.
Important signals like:
can be preserved intentionally.
Instead of scraping, cleaning, parsing, chunking, and embedding content themselves, AI platforms can consume ready-to-use knowledge artifacts.
Publisher-provided transformations come with clear attribution and traceability.
This is increasingly important as AI answers are scrutinized for accuracy and fairness.
Publishers also benefit significantly.
Instead of allowing third parties to summarize or reinterpret content, publishers define the canonical representation.
Affiliate links, product partnerships, and editorial positioning can be preserved in the knowledge representation.
Transformations themselves become licensable assets.
Examples include:
If AI systems can access clean semantic representations through a licensed interface, there is less incentive to crawl raw pages aggressively.
Some platforms attempt to solve the AI access problem by offering generic transformation services.
Examples include:
These solutions can be useful for basic access control.
But they suffer from a fundamental limitation.
They do not understand the publisher's content model.
Generic transformation systems cannot easily capture:
As a result, the outputs they generate are often lower quality.
AI systems must still reconstruct meaning from incomplete representations.
Peek-Then-Pay ties these ideas together.
It introduces a simple concept:
AI agents should be able to preview what a resource contains before licensing full access.
This preview might include:
If the resource is valuable, the agent can request the full licensed representation.
That representation might include:
The result is a marketplace where:
Each participant focuses on what they do best.
The architecture that emerges looks like this:
Licensing → centralized policy and agreements
Usage accounting → distributed assertion validation and local metering
Content transformation → publisher-controlled semantic representation
This separation of concerns enables:
Most importantly, it aligns incentives.
Publishers want their knowledge represented accurately.
AI platforms want reliable, high-quality sources.
Peek-Then-Pay provides a framework where both sides win.
As AI systems become the primary interface to information, the way knowledge is accessed will continue to evolve.
The web was built for humans reading HTML.
AI systems need something different.
They need:
The FetchRight / Peek-Then-Pay model enables this future by recognizing that licensing, metering, and transformation are separate problems that deserve separate solutions.
When these concerns are cleanly separated, the ecosystem becomes healthier for everyone involved.
Publishers retain control of their voice and intellectual property.
AI platforms gain access to higher-quality knowledge.
And the web evolves into a more sustainable knowledge infrastructure for the AI era.
Publishers must now serve human comprehension and machine reasoning simultaneously. The challenge is not choosing one audience over the other; it is designing content ecosystems that preserve epistemic integrity across both.
The most consequential flaw in today's AI content ecosystem is architectural. Retrieval systems collapse discovery and access into a single step, inverting the sequence that made the open web governable.
FetchRight and Peek-Then-Pay let LLMs ask publishers directly for pre-filtered, attribution-ready context — cutting duplicate embedding work and lowering costs for both sides.