Comment by DeusExMachina
1 hour ago
I don't understand the endgame here. Websites let Google crawl their content in exchange of traffic. If Google cuts that out completely, what incentive do websites have to not block the Google crawlers?
I understand that Google is feeling an existential threat from other AI products that provide answers directly. But they must also understand their symbiotic relationship with the web.
The end game is the consumer no longer leaving Google and the web becoming synonymous to Google for them. Why shop on some random website when you can have Gemini buy it for you? Why look for information on Wikipedia when… you get the idea.
I think the coming years will be pivotal for the web. Facebook attempted a similar strategy back when their apps got traction, but they ultimately failed. Let’s hope Google fails too.
We're going back to the CompuServe/AOL/Prodigy model
The long-run doesn't matter as much as the short-term gains for those in power.
What I really don't understand is where the next generation of training material will come from. If websites stop being published and/or crawled, how will the machine continue to be fed.
Current executives think it's a problem for the future executives.
They have enough internet slop. The training material they care about comes from experts, not randos online. This is why Mercor and Scale are billion dollar companies.
Either Google is ignoring that, or crossing their fingers and hoping that one LLM can produce data to train another one.
The web is going to become China, which is a collection of walled gardens
Google ignores robots.txt and botnets residential addresses to crawl anyway? (LLM startups already do this.)
> If Google cuts that out completely, what incentive do websites have to not block the Google crawlers?
Completely, yes, that destroys the incentive. But they can reduce it 80% or 90% or so, to the point that it's just barely worthwhile to allow their crawlers.
You will be kept inside the Google ecosystem the same way people are kept inside Facebook.
I’m curious how they plan to generate new content in the future, because it seems obvious that simple web pages will become obsolete and eventually stop being filled with fresh data.
It will probably end with a warning every time you click a link, something like: “You are leaving to an external unsafe site.”
The impression I get from Google's own marketing material is that Google doesn't believe in "the web". And it hasn't believed in the web for years.
Think about it. Pretty much every time they show a search box with someone asking for directions to reach a physical place, what hours is it open, etc.
The greatest thing about the internet is that it has removed distances around the whole world, but Google's major value proposition seems to be that... it can accurately index and query information about local businesses?