Jarrett Sidaway
CEO & Co-Founder, FetchRight
Artificial intelligence has outgrown the infrastructure of the early web. The systems that once governed search, indexing, and content access were built for a world where humans clicked links, navigated pages, and read articles one at a time. But today, the dominant consumers of publisher content are not human readers — they are AI crawlers, retrieval systems, and generative models.
These systems operate differently. They need structured signals, clear permissions, machine-readable terms, and predictable pathways for access and use. They require clarity and consistency so they can consume content responsibly and produce high-quality answers.
Yet publishers and AI platforms are still interacting within a framework defined by robots.txt, scattered paywalls, and reactive legal efforts. It is no wonder the current environment feels chaotic. There is no shared language, no common protocol, no reliable handshake that governs how content should flow between publishers and AI.
That is why a new standard is needed — one built intentionally for the AI era, not inherited from the early web.
Peek-Then-Pay, an open licensing standard developed collaboratively and deployed through FetchRight, aims to be that handshake.
Crawlers arrive at publisher domains today with very few signals to guide them. Some are legitimate AI agents attempting to comply with rules. Others are intermediaries scraping on behalf of larger systems. Many operate in a gray zone: partially compliant, partially opaque, and increasingly sophisticated.
Without a modern protocol that governs access, the environment breaks down in predictable ways.
For publishers, the lack of structure means:
For AI platforms, the lack of structure means:
For consumers, the lack of structure means:
The friction, inefficiency, and risk in the system do not result from malicious intent. They result from missing architecture.
We have never built a formal, enforceable system for how content and rights should flow between publishers and machines.
Peek-Then-Pay changes that.
Peek-Then-Pay establishes a framework for transparent, enforceable collaboration. It introduces a consistent, machine-readable protocol for how AI systems discover, evaluate, and license publisher content.
The core ideas are simple, but transformative:
1. AI receives a structured "peek" — not a blind crawl.
Instead of scraping or guessing, AI agents receive a preview of the content under defined terms. This preview includes the essential signals needed to evaluate whether deeper access is valuable: content type, pricing, rights, and format.
2. Publishers set the rules of engagement in a standardized manifest.
Peek-Then-Pay uses a predictable file — peek.json — where publishers declare how they want AI to interact with their domain: what can be accessed, what requires a license, and what formats are available.
3. Licensing becomes automatic, not manual.
If the crawler wants full access or specialized formats, it can license the content directly through a structured and auditable process — the "pay" step. This removes the endless negotiation cycles that have slowed progress between publishers and AI platforms.
4. Usage becomes verifiable and transparent.
AI platforms receive clean, compliant content aligned to their needs. Publishers receive tracking, reporting, and compensation tied to actual usage.
5. Collaboration becomes scalable.
Instead of one-off deals with a handful of large platforms, Peek-Then-Pay creates a system where publishers can interact with AI at scale, across the wider ecosystem.
This is not a paywall.
It is not a content blockade.
It is a structured, mutually beneficial exchange — a modern protocol for a modern ecosystem.
A sustainable ecosystem requires aligned incentives. Right now, publishers and AI platforms are misaligned not because their goals are incompatible, but because the infrastructure cannot support their shared interests.
Peek-Then-Pay aligns those interests in three critical ways.
1. Publishers regain agency — without blocking innovation.
Publishers decide:
This turns them from passive sources of ingestion into active participants shaping how their authority is accessed.
2. AI platforms get higher-quality content, faster and more efficiently.
AI systems benefit enormously from:
This reduces compute costs, improves accuracy, and builds trust with users.
3. Consumers get answers rooted in verified expertise.
Users don't see the handshake — but they experience its value. When AI systems use structured, rights-cleared content, answers become:
Everyone wins in a structured ecosystem.
When access becomes structured and licensing becomes automatic, a new set of opportunities emerges — opportunities that are impossible in the current environment.
Richer AI experiences built on verified expertise
Publishers can provide not just full articles, but structured summaries, fact panels, Q&A fragments, definitions, and domain-specific interpretations. AI platforms can build deeper, more accurate responses using the publisher's own framing and voice.
Content tailored to the use case
Instead of treating all content as interchangeable, Peek-Then-Pay enables the delivery of content optimized for:
Publishers can define what version of their content is used in each scenario.
New monetization pathways for publishers
Licensing is not limited to full-article retrieval. Publishers can monetize:
This is a new economic layer, not a rehash of existing models.
Lower compliance and legal exposure for AI platforms
When access is rights-cleared and transparent, AI companies can innovate faster without the looming threat of lawsuits or undefined liability.
This is the model that unlocks forward progress.
If this system were proprietary, it would simply recreate the silos that have fractured the media economy for years. Peek-Then-Pay is explicitly open because AI is a network-wide phenomenon. Publishers need consistency across platforms, and AI platforms need a reliable way to comply across domains.
An open standard:
It creates a shared foundation for collaboration — a common handshake recognized across the modern web.
The relationship between publishers and AI systems cannot be governed by the assumptions of the early internet. Crawlers, models, and answer engines require modern rules for engagement — rules that protect publisher authority, reduce friction for AI platforms, and produce better outcomes for consumers.
Peek-Then-Pay provides the framework this era requires: structured discovery, transparent licensing, and aligned incentives. It replaces guesswork with clarity and replaces extraction with collaboration.
The future of AI will not be defined by who controls distribution — it will be defined by who controls the quality, structure, and terms of the content that powers it. Publishers hold that authority today. Peek-Then-Pay ensures they can exercise it.
How the Peek-Then-Pay standard is reshaping the relationship between AI systems and content creators, giving publishers control while enabling efficient AI development.
FetchRight and Peek-Then-Pay let LLMs ask publishers directly for pre-filtered, attribution-ready context — cutting duplicate embedding work and lowering costs for both sides.
Every AI query burns thousands of tokens just to reacquire context. FetchRight and Peek-Then-Pay offer a smarter approach where publishers provide structured data and LLMs pay for efficiency, not redundancy.