IAB Europe unveils framework for AI publisher compensation

According to IAB Europe Data Analyst Dimitris Beis, the framework addresses "a paradigm of publisher remuneration for content ingestion" through three core mechanisms: content access controls, discovery protocols, and monetization APIs. The 11-page document establishes technical specifications for AI platforms accessing publisher content.

The framework emerges from documented traffic disruptions affecting digital publishers. According to Similarweb data cited in the report, referrals from AI platforms increased 357% year-over-year in June 2025, reaching 1.13 billion visits compared to 191 billion visits from organic Google search. However, news and media sectors experienced 770% traffic growth from AI platforms during the same period.

Cloudflare CEO Matthew Prince, speaking at a Cannes event, described shifting economics in content crawling. According to the framework, Prince reported the ratio of pages crawled to visitors referred increased from 2:1 a decade ago to 6:1 at the beginning of 2025 and 18:1 in June. OpenAI's ratio reportedly grew from 250:1 to 1,250:1 during this timeframe.

The framework contradicts Google's August rebuttal claiming stable year-over-year referrals from organic search. According to Chartbeat research covering 565 US and UK news websites, search referral consistency has been maintained over the past year. Google acknowledged certain query types may not generate clicks, similar to previous features like sports scores.

Adobe research conducted between July 2024 and February 2025 revealed AI-referred visitors stayed 8% longer on sites, viewed 12% more pages, and showed 23% lower bounce rates. However, these visitors lagged 9% behind non-AI-referred users in conversion rates.

The IAB framework proposes blocking unauthorised scraping through robots.txt files and Web Application Firewall methods. According to the document, unauthorised scraping increased 40% from Q3 to Q4 2024, with robots.txt compliance declining significantly.

Three content discovery mechanisms form the framework's second component. Publishers would implement content access rules pages containing usage terms, scraper instructions, contact information, and content metadata. JSON-based content metadata would provide site summaries and IAB content taxonomy mappings. An llms.txt markdown file would contain information digestible by large language models.

The monetization component introduces Cost-per-crawl (CPCr) APIs featuring tiered pricing based on content type, bot classification, and access frequency. According to the framework, a more sophisticated LLM ingest content API would support per-query pricing through bid-response exchanges, enabling real-time content valuation.

The per-query model addresses retrieval-augmented generation, where AI platforms query publisher content directly rather than using pre-trained datasets. According to the document, this approach "more closely tracks value extracted from using publisher content and facilitates a fairer deal than cost-per-crawl."

The framework identifies three implementation challenges. Controlling content access requires commitment from AI operators beyond technical measures, as multiple investigations suggest robots.txt compliance varies significantly. Auction dynamics differ from advertising markets, with single AI operators typically bidding rather than multiple competing buyers.

Content valuation presents complexity in determining marginal benefits of additional content for LLM responses. According to the framework, pricing decisions become probabilistic when based solely on metadata, potentially requiring verification mechanisms before content licensing.

Alternative models include revenue-sharing subscriptions, where Perplexity distributes 80% of user fees to participating publishers based on engagement metrics. Bilateral licensing agreements between major publishers and AI platforms provide direct compensation but concentrate benefits among large content creators.

Collective licensing schemes, similar to music rights societies, would create central compensation pools distributed according to usage measurements. According to the framework, this model requires regulatory action and allocation consensus.

The framework establishes three requirements for viable compensation models. Effective content access control must reliably block unauthorised scraping. Purpose-limited use assurance prevents single-query content from training dataset repurposing. Transparency in pricing and trade-offs provides publishers visibility into content usage and valuation.

Current conditions fail to meet these requirements. According to the document, unauthorised scraping continues rising as the root cause of publisher concerns. Most publishers lack visibility into content usage after access, with only large publishers securing protections through bespoke AI operator agreements.

Cloudflare recently introduced AI crawler blocking capabilities and piloting systems where AI platforms declare content access purposes while publishers control permissions. According to the framework, the company develops signed requests and mTLS technologies for strengthening crawler identification.

IAB Tech Lab CEO Tony Katsur has advocated for regulatory intervention, urging publishers to advocate for their interests. According to the document, structural solutions enforcing access control, transparency, and verifiable usage represent prerequisites before remuneration models can function at scale.

The marketing community faces significant implications from these developments. Publishers experiencing declining traffic revenues must evaluate alternative monetization strategies beyond traditional advertising models. AI-powered search features reduce click-through rates while maintaining content dependency for training and inference processes.

Campaign strategies may require adaptation as zero-click searches increase and publisher content appears in AI summaries without corresponding traffic. Performance measurement frameworks need updating to account for content usage in AI responses rather than website visit metrics.

The framework represents industrywide momentum toward formalised compensation structures. According to the document, remuneration models likely diverge rather than converge on single mechanisms, with publishers anticipating patchwork approaches depending on market position and jurisdiction.

IAB Europe's Artificial Intelligence Working Group seeks European publisher collaboration. The working group can be contacted through Dimitris Beis at beis [at] iabeurope [dot] eu for participation information.

Timeline

• March 2025: Google announces AI Overviews upgrades and experimental AI Mode
• May 2025: Brazilian journalism organizations request competition authority investigation into Google practices
• May 21, 2025: News Media Alliance calls Google AI features "theft"
• July 1, 2025: Cloudflare launches pay-per-crawl service in private beta
• July 30, 2025: IAB Tech Lab summit with 80+ media executives addresses AI scraping
• August 20, 2025: IAB Tech Lab launches Content Monetization Protocols working group
• August 28, 2025: Cloudflare expands 402 payment protocol for AI crawler communication
• September 2025: IAB Europe publishes "Crawling for Compensation" framework document
• September 2025: Anthropic settles $1.5 billion copyright lawsuit with authors

Summary

Who: IAB Europe Data Analyst Dimitris Beis authored the framework. The initiative involves publishers, AI platforms, and the IAB Tech Lab working group seeking European publisher collaboration.

What: A technical framework establishing three mechanisms for AI platform compensation to publishers: content access controls, discovery protocols, and monetization APIs including Cost-per-crawl and LLM ingest content APIs.

When: Published in September 2025, following industry discussions throughout 2025 including the July IAB Tech Lab summit and August working group launch.

Where: The framework applies globally but emphasises European implementation through IAB Europe's Artificial Intelligence Working Group collaboration with European publishers.

Why: Addresses declining publisher revenues from increased AI content scraping (357% growth year-over-year) and zero-click searches (rising from 56% to 69% in May 2025) while establishing fair compensation for content used in AI training and inference.