Comment by ajross
3 days ago
What's interesting to me is that this kind of behavior -- slightly-buffleheaded synthesis of very large areas of discourse with widely varying levels of reliability/trustworthiness -- is actually sort of one of the best things about AI research, at least for me?
I'm pretty good at reading the original sources. But what I don't have in a lot of cases is a gut that tells me what's available. I'll search for some vague idea (like, "someone must have done this before") with the wrong jargon and unclear explanation. And the AI will... sort of figure it out and point me at a bunch of people talking about exactly the idea I just had.
Now, sometimes they're loons and the idea is wrong, but the search will tell me who the players are, what jargon they're using to talk about it, what the relevant controversies around the ideas are, etc... And I can take it from there. But without the AI it's actually a long road between "I bet this exists" and "Here's someone who did it right already".
Yeah, this is by far the biggest value I've gotten from LLMs - just pointing me to the area of literature neither me nor any of my friends have heard of, but which have spent a decade running about the problems we're running into.
In this case, all that matters is that the outputs aren't complete hallucination. Once you know the magic jargon, everything opens up easily with traditional search.
The issue is when the output is good sounding but complete fantasy.
I’ve had it ‘dream’ up entire fake products, APIs, and even libraries before.
And that becomes obvious when you go to look for it. These work best in situations where false positives have low cost/impact, but true positives are easily verifiable and have high impact. in other words, problems in epistemological p-space. :)