Comment by alightsoul

I see a new version of the web, web 4.0, being exactly this. Web 4.0 is the source of the live data. Web 4.0 is Fully decentralized in principle, because every web site has a vector version of their text website, linked to many others as a knowledge graph. The knowledge graph's links are like hyperlinks.

The vector databases might be free or paid or have ads. There is no difference between web 4.0 and an ai model that has all of it as its live data.

It is up to every client to turn that data into ai responses so in theory anyone would instantly have frontier scale ai which sounds as crazy as everyone having a computer, and the internet and the www in the 1980s.

It also, by coincidence, fully fulfils the goals of web 3.0 aka the semantic web.

Moderation of content and ranking content like a search engine would be left to the client. Or an instance like a fediverse mastodon instance or atproto with bluesky. There will be a few large models like web browsers or operating systems or social networks or fediverse instances today that will compile web 4.0 into a result in an ai agent like say Hermes and perform content moderation and ranking. You could in theory use a decensored model to perform crime like you can with the web today with Tor like selling drugs or piracy, with search engines that are not Google or defederated mastodon instances

Humans can't read raw html easily and can't read vector databases easily either. It would be read by an ai model connected to an ai agent, like a rendering engine connected to a graphical user interface

It might need a new protocol from the IETF and a standards body like the w3c? And a web server could automatically vectorize content via embeddings and link them in a decentralized knowledge graph for either graph rag after undoing vectorization or like this paper suggests being connected directly to the vectors themselves. The embeddings would be decentralized too or the website could offer them and there could be a way to translate between them

Web crawling would die and so would search engines because they would move to the user with their own ai compute and ai agent and data would already be machine readable and queryable. The data is always online and queryable programmatically so there would be no need for offline data silos. Unlike mcp servers which already do this, the data is not text based and there is no search engine to link pages to queries because the knowledge graph would be exactly that. The knowledge graph would be the new version of DNS for translation of a query into data like ip addresses into web domains because knowledge graphs are what search engines already are in a way so I don't see how a search engine would be useful except for a human to query it directly as done currently