If a given fact is relatively new, I would expect (and forgive) search results to not necessarily reflect that.
But in this case we're talking about ~40 year old song. The reddit thread mentioned in the article is 10 months old.
I've also checked this logged in and incognito and get consistent (and seemingly correct) answers.
So what are we left with? Is the author in some experiment giving bad results? Were there bad results that got corrected between the time of writing and now? Did some Google search quality engineer see this and correct it?
All of these seem like a stretch. I'm perfectly willing to entertain the idea that Google has become worse but there are rarely conrete examples. It all just comes down to vibes. And when there are examples (as per this post), it doesn't match my experience. Can I really be that lucky? That seems to stretch credulity.
Probably the most egregious example of bad search results I've personally seen is how an astroturfed propaganda site was the top result for the Armenian genocide for literally years [1].
I feel like 'vibes' is being used to dismiss anecdotal evidence, which, amalgamated across an increasing swathe of technical users, approaches actual evidence.
The very fact that it was clearly wrong in the example shows you that Google is capable of building a flawed Knowledge graph. This is not vibes, this is either a bug or, frankly far more likely, the inevitable problem of trying to compress all of human existence into a LLM model.
Given that LLM training is an imperfect storage mechanism, is it really hard to believe that a given iteration of the model will just not "know" arbitrary facts.
My personal anecdata on this is that searching for the 'ephemeral port range' the Google AI summary was wrong, even though the Wikipedia reference it used was correct.
If a given fact is relatively new, I would expect (and forgive) search results to not necessarily reflect that.
But in this case we're talking about ~40 year old song. The reddit thread mentioned in the article is 10 months old.
I've also checked this logged in and incognito and get consistent (and seemingly correct) answers.
So what are we left with? Is the author in some experiment giving bad results? Were there bad results that got corrected between the time of writing and now? Did some Google search quality engineer see this and correct it?
All of these seem like a stretch. I'm perfectly willing to entertain the idea that Google has become worse but there are rarely conrete examples. It all just comes down to vibes. And when there are examples (as per this post), it doesn't match my experience. Can I really be that lucky? That seems to stretch credulity.
Probably the most egregious example of bad search results I've personally seen is how an astroturfed propaganda site was the top result for the Armenian genocide for literally years [1].
[1]: https://www.vice.com/en/article/how-google-searches-are-prom...
I feel like 'vibes' is being used to dismiss anecdotal evidence, which, amalgamated across an increasing swathe of technical users, approaches actual evidence.
The very fact that it was clearly wrong in the example shows you that Google is capable of building a flawed Knowledge graph. This is not vibes, this is either a bug or, frankly far more likely, the inevitable problem of trying to compress all of human existence into a LLM model.
Given that LLM training is an imperfect storage mechanism, is it really hard to believe that a given iteration of the model will just not "know" arbitrary facts.
My personal anecdata on this is that searching for the 'ephemeral port range' the Google AI summary was wrong, even though the Wikipedia reference it used was correct.