Comment by quickthrower2
4 years ago
This could be the tech that dislodges Google from its monopoly! There are definitely questions I would prefer this experience than Google and the inevitable SEO spam for introduction type questions. If the generated text included further links to learn more it would be awesome.
> I work at Alphabet and I recently went to an internal tech talk about deploying large language models like this at Google. As a disclaimer I'll first note that this is not my area of expertise, I just attended the tech talk because it sounded interesting.
> Large language models like GPT are one of the biggest areas of active ML research at Google, and there's a ton of pretty obvious applications for how they can be used to answer queries, index information, etc. There is a huge budget at Google related to staffing people to work on these kinds of models and do the actual training, which is very expensive because it takes a ton of compute capacity to train these super huge language models. However what I gathered from the talk is the economics of actually using these kinds of language models in the biggest Google products (e.g. search, gmail) isn't quite there yet. It's one thing to put up a demo that interested nerds can play with, but it's quite another thing to try to integrate it deeply in a system that serves billions of requests a day when you take into account serving costs, added latency, and the fact that the average revenue on something like a Google search is close to infinitesimal already. I think I remember the presenter saying something like they'd want to reduce the costs by at least 10x before it would be feasible to integrate models like this in products like search. A 10x or even 100x improvement is obviously an attainable target in the next few years, so I think technology like this is coming in the next few years.
https://news.ycombinator.com/item?id=33820750
https://blog.google/technology/ai/lamda/
In time, this tech is also likely to spark a new arms race, as happened with Google vs SEO spam.
Generated content as the new blogspam, in turn needing to be filtered out for training the next generation of models.
Maybe but the straight up incentive of tricking someone on your site wont be there, as well as a clear set of tactics. I feel like it would be as challenging as DDOSing cloudflare. It just might be too expensive to do outside of countries or large corps.
Yeah what would be the mechanism for it to decide what statements on random sites are True?
I'd imagine that it would be similar to the mechanism that is already used by existing search engines like Google, and people like you & I. So a combination of reputation & heuristics.
I think the new blog spam will be better than the average non-spam blog.
This XKCD strip is looking mighty prescient right now: https://xkcd.com/810/
> If the generated text included further links to learn more it would be awesome.
A new type of language model has been developed that takes a query as input and produces document IDs as output, similar to how Google works, but without relying on an inverted index of keywords. This model can be used on any text to obtain the most relevant references.
How would someone learn more about this language model? What is it called?
> This could be the tech that dislodges Google from its monopoly!
Google has been part of the arms race for this kind of AI from the beginning, so even if this kind of AI (alone or connected to other things) dislodges Google’s current model as an entry point for information queries, it may not dislodge Google from its position.
When Google simply began showing more data from indexed sites, it caused outrage because users wouldn’t visit & monetize. Here we have three guys who literally crawled the entire Internet into a black box to monetize.
So far we praise them for this like we praised Google, which thus makes me think they can in fact dislodge. Not sure I feel enthusiastic about it, however.
Google has been chasing this stuff from the beginning and has probably been devoting tons more resources to it than OpenAI. Given their infrastructure and existing user advantage, once they think this will be feasible and better than existing Google Search, they will make sure it becomes Google Search.
the problem here is that GPT3 is trained off the same data Google has access to. It also costs probably several orders of magnitude more per query, so there's no way to do it at scale affordably at the moment. ChatGPT is going to be put behind a paywall soon for that reason, it's an expensive marketing gimmick