Manage AI Access to Your Content with Google-Extended

Publishers work hard to craft awesome content that keeps their readers hooked, right? But in today’s digital world, it’s critical to ensure that content stays safe and under your control.

How can you protect your content from unwanted scraping and data collection in a world with generative AI?

At Mediavine, we believe that generative AI does have value for publishers, especially around speeding up workflows and potentially reducing costs. But there are definitely drawbacks and concerns, such as copyright infringement, that need to be taken seriously.

Google agrees, which is why it’s releasing Google-Extended, a solution to allow more control for publishers to protect their content.

Publishers want greater choice and control over how their content is used for emerging generative AI use cases—if it’s used at all. Google-Extended is Google’s solution to that concern. Google seeks to develop AI tools guided by AI principles that respect consumer privacy.

What’s Google-Extended?

Before we go any further, it’s important to state that this is not an official Mediavine recommendation but an available option that we’re presenting to you as your business partner.

Even though you have the option to use Google-Extended, it might impact your site’s search performance. It’s important to remember that every website is unique, so whether your traffic goes up or down will really depend on your specific site.

That said, Google-Extended allows publishers to control access to the content on their sites; website administrators can choose whether to help Google’s AI models or not.

Google-Extended is one of the ways Google is testing simple, scalable controls to provide transparency. The idea behind it is fairly straight-forward: as AI keeps on growing and is rolled out in more and more ways, the ad tech industry — like most industries — is only going to become more complex.

Google-Extended allows publishers to manage whether their sites will work with Google’s AI solutions, such as Bard and Vertex AI generative APIs (application programming interfaces), to improve them for accuracy and capability over time. Google-Extended also includes future generations of models that power those products.

How Can Publishers Use Google-Extended?

Now, for the technical stuff.

Google-Extended is a standalone product token that publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs. It’s also what Google considers to be a “common crawler,” which is sometimes referred to as a “robot” or “spider.”

A crawler is a generic term for any program that’s used to automatically discover and scan websites by following links from one web page to another.

Google uses crawlers to perform actions for its products, either automatically or when triggered by a user request. They help Google build and improve search indices, perform product-specific crawls and they’re useful for analysis.

They always obey robots.txt rules and generally crawl from the IP ranges published in the googlebot.json object.

So, how can Google-Extended be implemented on your site?

It’s time for that caveat again: this is not an official Mediavine recommendation, but an available option that we’re presenting to you as your business partner.

Visit Google’s Overview of Google crawlers and fetchers (user agents) to learn how to implement Google-Extended.

Conclusion

Google has introduced Google-Extended as one solution to empower publishers with greater choice and control over their content in the era of generative AI.

Implementing Google-Extended is an option for publishers, but individual outcomes may vary depending on the unique characteristics of your website.

We encourage publishers to explore Google-Extended and its potential benefits, keeping in mind that the choice to engage with AI technologies should align with your specific goals and objectives. Mediavine’s Support team isn’t equipped to assist our publishers with technical support for Google-Extended implementation, so for that, we recommend contacting your web developer. 

As your business partner, we’re dedicated to offering you valuable tools to navigate the ever-changing digital landscape. Google-Extended is one of those tools.