Cloudflare Gives Creators New Tool to Control Use of Their Content

The Internet is shifting from “search engines,” which provided a treasure map of links that a user could explore for information, to “answer engines” powered by AI, which give a direct answer without a user ever needing to click on the original site’s content. This severely threatens the original business model of the Internet, where websites, publishers, and content creators could earn money or fame by driving traffic and views to their site. Today, AI crawlers scrape vast troves of data from websites, but website operators have no way to express the nuances of whether, how, and for what purpose they may want to allow their content to be used. Robots.txt files allow website operators to specify which crawlers are allowed and what parts of a website they can access. It does not, however, let the crawler know what they are able to do with the content after accessing it. There needs to be a standard, machine-readable way to signal how data can be used even after it has been accessed.

“The Internet cannot wait for a solution, while in the meantime, creators’ original content is used for profit by other companies,” said Matthew Prince, co-founder and CEO of Cloudflare. “To ensure the web remains open and thriving, we’re giving website owners a better way to express how companies are allowed to use their content. Robots.txt is an underutilized resource that we can help strengthen, and make it clear to AI companies that they can no longer ignore a content creator's preferences.”

Cloudflare believes that an operator of a website, API, MCP server, or any Internet-connected service, whether they are a local news organization, AI startup, or an ecommerce shop, should get to decide how their data is used by others for commercial purposes. Today, more than 3.8 million domains use Cloudflare’s managed robots.txt service to express they do not want their content used for training. Now, Cloudflare's new Content Signals Policy will enable users to strengthen their robots.txt preferences with a clear set of instructions for anyone accessing the website via automated means, such as an AI crawler. The policy will now inform crawlers by:

• Explaining how to interpret the content signals in simple terms: “Yes” means allowed, “no” means not allowed, and no signal means no expressed preference.
• Defining the different ways that a crawler typically uses content in clear terms, including search, AI input, and AI training.
• Reminding companies that website operators’ preferences in robots.txt files can have legal significance.

While robots.txt files may not stop unwanted scraping, Cloudflare’s aim is that this improved policy language will better communicate a website owner’s preferences to bot operators, and drive companies to better respect content creator preferences.

Starting today, Cloudflare will automatically update the robots.txt files to include this new policy language for all customers that ask Cloudflare to manage their robots.txt file. For anyone who wants to declare how crawlers can use their content via customized robots.txt files, Cloudflare is publishing tools to help.

Organizations have seen the need for solutions like the Content Signals Policy, as a way to offer more direction over how their content is used:

• News/Media Alliance: “We are thrilled that Cloudflare is offering a powerful new tool, now widely available to all users, for publishers to dictate how and where their content is used. This is an important step towards empowering publishers of all sizes to reclaim control over their own content, and ensure they can continue to fund the creation of quality content that users rely on. We hope that this encourages tech companies to respect content creators’ preferences. Cloudflare is showing that doing the right thing isn’t just possible, it’s good business.” – Danielle Coffey, President and CEO of the News/Media Alliance
• Quora: “We applaud CloudFlare's leadership and support their efforts in building controls and protocols to help publishers manage how their content is accessed.” – Ricky Arai-Lopez, Head of Product at Quora.
• Reddit: “For the web to remain a place for authentic human interaction, platforms that empower communities must be sustainable. We support initiatives that advocate for clear signals protecting against the abuse and misuse of content.” – Chris Slowe, CTO of Reddit
• RSL Collective: “We are excited to partner with Cloudflare on the launch of the Cloudflare Content Signals Policy, an essential step forward in allowing publishers to assert their rights and clearly define how companies may use their content. The open RSL standard, developed in cooperation with the Internet’s leading publishers, is designed to complement the Content Signals protocol by enabling content owners to not only protect their rights, but also define machine-readable licensing and compensation terms for those use cases. Together, the RSL Collective and Cloudflare are advancing a shared vision: a sustainable open web where publishers and creators thrive and are fairly compensated by AI companies.” – Eckart Walther, co-founder of the RSL Collective
• Stack Overflow: “The nature of the Internet and its implicit agreement with content publishers has changed quite dramatically over the past couple of years. With our large corpus of ~70 billion tokens of data, Stack Overflow is proud to partner with the leading AI labs and cloud providers on the data licensing front and we applaud Cloudflare for playing a central role to empower and protect content creators to build a scalable system for the internet in this new AI era.” – Prashanth Chandrasekar, CEO of Stack Overflow

To learn more, please check out the resources below:

• Blog: Giving Users Choice with Cloudflare’s New Content Signals Policy
• http://contentsignals.org/

About Cloudflare

Cloudflare, Inc. (NYSE: NET) is the leading connectivity cloud company on a mission to help build a better Internet. It empowers organizations to make their employees, applications and networks faster and more secure everywhere, while reducing complexity and cost. Cloudflare’s connectivity cloud delivers the most full-featured, unified platform of cloud-native products and developer tools, so any organization can gain the control they need to work, develop, and accelerate their business.

Powered by one of the world’s largest and most interconnected networks, Cloudflare blocks billions of threats online for its customers every day. It is trusted by millions of organizations – from the largest brands to entrepreneurs and small businesses to nonprofits, humanitarian groups, and governments across the globe.

Learn more about Cloudflare’s connectivity cloud at cloudflare.com/connectivity-cloud. Learn more about the latest Internet trends and insights at https://radar.cloudflare.com.

Follow us: Blog | X | LinkedIn | Facebook | Instagram

Forward-Looking Statements

This press release contains forward-looking statements within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended, which statements involve substantial risks and uncertainties. In some cases, you can identify forward-looking statements because they contain words such as “may,” “will,” “should,” “expect,” “explore,” “plan,” “anticipate,” “could,” “intend,” “target,” “project,” “contemplate,” “believe,” “estimate,” “predict,” “potential,” or “continue,” or the negative of these words, or other similar terms or expressions that concern Cloudflare’s expectations, strategy, plans, or intentions. However, not all forward-looking statements contain these identifying words. Forward-looking statements expressed or implied in this press release include, but are not limited to, statements regarding Cloudflare’s plans and objectives for its Content Signals Policy, the capabilities and effectiveness of Cloudflare’s Content Signals Policy and Cloudflare’s other products and technology, the benefits to Cloudflare’s customers from using Cloudflare’s Content Signals Policy and Cloudflare’s other products and technology, Cloudflare’s global network, Cloudflare’s products and technology, Cloudflare’s technological development, future operations, growth, initiatives, or strategies, and comments made by Cloudflare’s CEO and others. Actual results could differ materially from those stated or implied in forward-looking statements due to a number of factors, including but not limited to, risks detailed in Cloudflare’s filings with the Securities and Exchange Commission (SEC), including Cloudflare’s Quarterly Report on Form 10-Q filed on July 31, 2025, as well as other filings that Cloudflare may make from time to time with the SEC.

The forward-looking statements made in this press release relate only to events as of the date on which the statements are made. Cloudflare undertakes no obligation to update any forward-looking statements made in this press release to reflect events or circumstances after the date of this press release or to reflect new information or the occurrence of unanticipated events, except as required by law. Cloudflare may not actually achieve the plans, intentions, or expectations disclosed in Cloudflare’s forward-looking statements, and you should not place undue reliance on Cloudflare’s forward-looking statements.

© 2025 Cloudflare, Inc. All rights reserved. Cloudflare, the Cloudflare logo, and other Cloudflare marks are trademarks and/or registered trademarks of Cloudflare, Inc. in the U.S. and other jurisdictions. All other marks and names referenced herein may be trademarks of their respective owners.

View source version on businesswire.com: https://www.businesswire.com/news/home/20250924718008/en/