Google today announced a new “standalone product token,” Google-Extended, that lets you control whether Bard and Vertex AI can access the content on your site. This seems to be the end result of a “public discussion” Google initiated in July, when the company promised to gather “voices from across web publishers, civil society, academia and more fields” to talk about choice and control over web content. Bard is Google’s conversational AI tool. Vertex AI is Google’s machine learning platform for building and deploying generative AI-powered search and chat applications. The announcement. In a blog post, Google said:
What is Google-Extended. Google calls it “A standalone product token that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products.” The new crawler has been added to the Google Search Central documentation on web crawlers. What Google is saying. The company said Google-Extended gives publishers “choice and control”:
Robots.txt. You can use robots.txt to block Google-Extended from accessing your content, or parts of it. To fully block Google-Extended, add the following to your site’s robots.txt:
Why we care. We know 242 of the most popular 1,000 websites have already decided to block GPTBot, OpenAI’s web crawler, since it launched in August. Now you can decide whether your website should opt out of helping Google improve its AI products. Is this the right answer? In Robots.txt is not the answer: Proposing a new meta tag for LLM/AI, Search Engine Land contributor argued why using robots.txt for managing data usage in LLMs is the wrong approach. Seems Google didn’t agree. Dig deeper. Crawlers, search engines and the sleaze of generative AI companies The post Google introduces Google-Extended to let you block Bard, Vertex AI via robots.txt appeared first on Search Engine Land. via Search Engine Land https://ift.tt/esxNyPl
0 Comments
Leave a Reply. |
Archives
April 2024
Categories |