Google-Extended Is The New Google Crawler To Block Bard Or Google AI - Via Search Engine Roundtable

Vivek Sharma -

 

Google-Extended Is The New Google Crawler To Block Bard Or Google AI
Image Seouce: seroundtable.com

Google has announced a new Googlebot, a new Google crawler, named Google-Extended that you can use to control if your content can help improve Bard and Vertex AI generative APIs or future Google AI products. So if you want to disallow Bard from using your content, you specify so in your robots.txt with the user agent Google-Extended.

Google won't crawl from Google-Extended, Google will still crawl from its normal Googlebot or other bots. But using Google-Extended will communicate to Google not to use that content for Bard or other AI Google projects. A Google spokesperson told me, "Google-Extended will tell Google not to use the site's content for Bard and Vertex AI generative APIs." "For Search, website administrators should continue to use the Googlebot user agent through robots.txt and the NOINDEX meta tag to manage their content in search results, including experiments like Search Generative Experience," Google added.

Essentially this allows you to allow Google Search to crawl, index and rank your website but disallow Bard or other Google AI projects from using your content.

This comes after Bing offered controls to block Bing Chat AI from using your site a week ago.

"Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products. By using Google-Extended to control access to content on a site, a website administrator can choose whether to help these AI models become more accurate and capable over time," Google wrote.

Google-Extended is a "standalone product token that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products," Google explained.

The User agent token is Google-Extended

"Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity," Google added.

I am not sure if this is the alternative approach for robots.txt for AI...

Note, Google News bot also works a similar way, where it does not crawl but uses the directive for using that content in Google News:

Forum discussion at X.

Blog Credit: seroundtable.com

Originally Published at: https://www.seroundtable.com/google-extended-crawler-ai-bard-36130.html


Popular Posts

A Complete Guide for SEO in 2019

Vivek Sharma -
Image
The Definitive Guide To SEO In 2019 This is the ultimate guide to SEO in 2019. And let me be clear about something: This is NOT a lame “SEO in 2019” predictions post. Instead, you’re going to see tested strategies that are working right now… and will work even better in 2019. So if you’re looking to up your SEO game this year, you’ll love this guide. Let’s dive right in. CHAPTER 1: RankBrain & UX Signals RankBrian and User Experience Signals User experience signals were HUGE this year. And I expect them to be even more important in 2019. In fact, Google previously announced that RankBrain was their third most important ranking factor: “In the few months it has been deployed, RankBrain has become the third-most important signal contributing to the result of a search query.” The question is: What is RankBrain, exactly? And how can you optimize for it? Google RankBrain: a Dead-Simple Explanation RankBrain  is a machine
Read full blog »

104 Helpful SEO Questions & Answers for Interview

Vivek Sharma -
Image
Discover all you need to know with these latest SEO questions and answers and embark on a successful SEO journey! Keep digging! 1. What is SEO & why is it so important? Ans.  In simple words, SEO or Search Engine Optimization denotes to any activity performed for the improvement of Search Engine rankings of websites, products, services or other content. It shows unpaid results which is also referred to as “free”, “organic”, “natural” or “earned” results. The importance of SEO starts with the desire of the companies to gain more traffic for their websites. The ranking over search engines matter because users pay more heed to the first 5 searches on Google. Moreover, the users tend to trust Google’s refined search results because they consider these searches to be more authentic and specific. 2. What is a Search Engine? Ans.  A search engine is a web-based software system which is developed to search and locate relevant information on the World Wide Web. Search engi
Read full blog »

How To Add Schema Markup To Your Wix Website?

Vivek Sharma -
Image
Adding Schema.org Markup Schema.org creates a structured data markup schema, which is supported by most major search engines. Adding schema markup can help search engines display a richer snippet of your pages in search results. It does so by providing search engines with additional information about the page content, which is then displayed beneath the page title in search results.  Click  here  to read more about Schema.org. Step 1 | Generate your schema markup Some of your site's content already contains schema markup, such as blog posts and your Wix Stores pages. To generate schema markup for other pages of your site we recommend using  this tool . You may also choose to use  Google's markup generator .  Step 2 | Add your schema markup to your Wix site 1  Sign in to your Wix account. 2  Go to your  Site Manager .  3  Click Tracking & Analytics.  4  Click + New Tool and select Custom from the dropdown.  5  Set up your Schema.org code: 5  Ente
Read full blog »