NEW YORK—As media companies face growing problems with AI bots and crawlers, the IAB Tech Lab has issued guidance that is designed to help publishers and content owners better evaluate and manage how non-human agents, including AI systems, access their content as they work to adopt the Tech Lab’s CoMP API framework for managing content crawling.
The new guidance, which is now available for public comment until June 26, 2026, outlines different approaches content owners and publishers can take to inform access from non-human user agents with the aim of helping the ecosystem make more informed decisions alongside the adoption of the CoMP API V1.
“Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way,” said Anthony Katsur, CEO, IAB Tech Lab. “This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace.”
The IAB Tech Lab noted that following the release of the CoMP API V1, it became clear that many content owners and publishers have not yet developed formal strategies around their approach to bots and crawlers, which sit outside the scope of the API itself.
To address this gap, the CoMP Working Group developed this guidance to outline the range of available approaches and the advantages and trade-offs of each strategy. This gives stakeholders a clearer understanding of how to manage access from all non-human traffic, including AI systems in a way that aligns with their business goals.
More specifically, the guidance is designed to support the adoption of the CoMP API by helping content owners navigate an area that can be operationally complex and resource-intensive and is aimed at non-technical leadership to give them the questions they need to ask.
Those questions will ultimately help them understand the value and hard costs associated with each lever they pull. With clearer direction on these strategies, organizations can make more confident decisions, reduce risk, lower their operating cost, and operate more efficiently as they engage with AI systems. Establishing these strategies is also critical to enabling partnerships between AI systems and content owners, which are necessary to build a sustainable and scalable content marketplace.
A number of industry participants involved in the development of the guidance also emphasized the importance of clarity and practical direction as the ecosystem evolves.
“Content owners are being asked to make important decisions quickly, often without clear frameworks,” said Shailley Singh, EVP & COO, IAB Tech Lab. “This guidance helps break down the options so they can choose an approach that aligns with their goals.”
“The AI revolution has caught many publishers with their front doors wide open, essentially leaving the lights on for bots and crawlers to pilfer intellectual property.” added Scott Messer, principal & founder, Messer Media, “This IAB Tech Lab guidance is the essential roadmap content owners need—a practical framework to navigate the complex maze of decisions, tech, and tools available to protect their business. By empowering publishers to identify who is really there, this guide helps them distinguish between a value-driving 'ally' and a resource-draining 'extractor'. Managing AI bots is no longer just a technical background task; it is a critical business strategy for protecting what you’ve built and restoring a fair exchange of value for your content.”
Similarly, Dave Bellous, vice president of strategy, Metal Toad, explained that “non-human traffic now dominates most of the open web, and AI agents and crawlers are steepening the curve. Blanket bot-blocking is no longer a useful strategy. For publishers, advertisers, and ad-tech alike, distinguishing beneficial automation from invalid traffic through graduated, standards-based controls has become a baseline requirement. The Bot and Crawler Guidance from IAB is a much-needed tool in crafting a modern web strategy.”
Tom Koch, head of sales, ads and marketing ecosystem at TwelveLabs also highlighted the importance of the guidance in addressing the problem of bots and crawlers: "As AI agents increasingly serve as proxies for consumers, publishers and platforms need clearer frameworks for transparency and control. The crawler management framework is an important step toward giving content owners greater visibility into how their content is accessed and used. TwelveLabs is proud to contribute as this work is foundational to ensuring video, the richest form of data, is no longer overlooked in how AI engages with the web."
The guidance was developed by the CoMP Working Group, with public comment providing an opportunity for industry feedback ahead of finalization. IAB Tech Lab will continue working with members to support implementation and encourage adoption of both the guidance and the CoMP API.
IAB Tech Lab invites the industry to participate in the public comment period until June 25, 2026. To review the guidance, https:///iabtechlab.com/botmanagement/.
