Comment by Hugsbox
3 hours ago
Yesterday I tried to google "can the Halifax Wanderers still make the CPL playoffs?"
So obviously what appears right at the top is the AI summary, which told me "they've already secured their #4 position and made the playoffs". I knew this wasn't true, and I guess I could have just scrolled down a bit further and found my answer but now I was curious.
So I said "that's not true, they're still #5, what I want to know is _could they still make the playoffs_"
It says they've got an upcoming game against Ottawa, and if they win their chances are good. That game has already taken place, so I correct it again and finally I get a reasonable answer.
My question is: what's the point of the AI in the search engine if it itself isn't going to use the search engine first before answering? Like, I can't wrap my head around that. The answer is on the same page as its hallucination. It could have done a cursory look around before first hallucinating something completely false, and when corrected the first time giving me outdated information. It's meant to be A SEARCH ENGINE!
This is similar to how, not too long ago, LLM's had extreme difficulty counting the number of letters in some words. LLM's don't "think" or "reason" in the normal definition of those terms. They can do some pretty amazing things, but still screw up basic things like telling you something that is obviously wrong and contradicts the top search results.
LLM's, in their present stage of development, are sort of like a crack-addled idiot savant. Sometimes they are obviously insane, and sometimes they seem quite cogent, but you must never trust them implicitly. This may be why they are so difficult to constrain. You could give them something equivalent to the laws of robotics, but following laws requires thought processes they simply don't have.
I'm actually sort of amazed Google doesn't make people accept some kind of butt-covering EULA and post disclaimers about the inaccuracy of results before even showing you their AI's output. Are they not being sued over this kind of thing?
[delayed]
LLMs still can't do math nor count letters in words. Nothing has changed there.
"LLMs can't do math" is a pretty hot take in September 2026.
8 replies →
ChatGPT live mode still hallucinates letters in words like this. HuskIRL and FatherPhi on youtube have done some hilarious videos with it in the last couple of weeks. Beyond miscounting the Rs in strawberry, ChatGPT will say there are two Ds in "your mom" and one D in "uranus" . I tried it myself to check that the videos weren't fake and sure enough it still has this failure mode.
Calling it a 'failure mode' implies it could be fixed. This is an inherent flaw in how LLMs work and will never go away until some new kind of architecture that can actually "read text" comes along.
3 replies →
> ChatGPT will say there are two Ds in "your mom" and one D in "uranus"
… Isn't it possible that it understands the innuendo and is going along with making the joke?
2 replies →
>Are they not being sued over this kind of thing?
Maybe but you have to have deep pockets just to get to the starting line. And then you need standing, and some injury to argue.
Corporations have been remarkably successful at arguing they are operating within the bounds of free speech, whether or not what is said is factual, and whether or not any fact checking has been done.
LLMs might not reason exactly like humans, but they do produce much better results if you turn reasoning on.
The "crack-addled idiot savant" phase was really circa 2024, before the big labs figured this out.
I think the issue here is that Google decided that doing reasoning in the AI overviews in Google search would be too slow (and probably also too expensive), so it's still stuck making 2024-era mistakes.
> This is similar to how, not too long ago, LLM's had extreme difficulty counting the number of letters in some words.
The specific issue of Google is that they are using an underpowered model, not fit to task, and much prone to hallucination than either OpenAI or Anthropic free tier offerings.
Google should at least match the frontier labs at the free tier (with some limit; after that, degrade quality), ffs
You're asking for something unreasonable. The number of Google searches per day is enormous and they haven't even been able to roll out AI overviews to everyone yet (they're missing in a new Firefox profile I just created). I wouldn't be surprised if the free tier frontier models cost over 100x more to serve than the AI overviews.
The specific issue is that search has become so bad that they think an LLM that gets answers wrong half of the time is a valid alternative, or, in fact, the "future" of search. Then they shoved that "alternative" to users with no way to disable it.
https://www.dw.com/en/german-court-holds-google-liable-for-f...
Google is no longer a search engine anymore. They don’t care about being one. Why you might ask?
First, a search engine indexes the web and makes it available to users. It’s been ages since Google did any of that. They no longer index sites or take ages to do so. Case in point, our cybersecurity startup (Webvetted.com) was launched in November 2025. Till date, only one page is indexed on the entire website. And I’ve talked to lots of other developers and it’s a common issue.
Secondly, a search engine organizes indexed information and makes it useful for people. Google is a basic LLM nowadays. They figured out that why organize and make the information useful when they could just answer the question with Gemini anyways? So they no longer bother to do the work of a search engine and are now just a lower-ranking open-source Chinese LLM
[dead]
I similarly noticed Gemini absolutely refuses to look at a url when I give it one and will instead just hallucinate based on what it thinks the url is. Here I am assuming Google will have the best web capabilities in its AI.
I searched for something, it told me that according to a YouTube video, the entire point of my search was wrong. I asked it for the source, I watched the video, it never made the claims Gemini hallucinated. I asked again and it claimed it scrubbed the video and found the point it made multiple times. I said those timecodes were wrong and it admitted it couldn't actually parse videos and just guessed.
What the fuck?
I don't understand how this isn't considered as active malice. Like purposefully outputting random stuff. Any other sort of computer system would get lot more flak than these are getting.
2 replies →
The scary (or funny) part is people use that for serious questions.
Typical exchange:
Me: "You bastard."
Gemini: "Fair callout. I should have been more up front that [has no idea what the fuck it is talking about]."
Same experience with the interaction I described in my comment... after I finally got the correct answer, I asked where it got the faulty information from. It said it had just simply fabricated it. That's actively worse than just saying "I don't know", for something that's sold to us as an easy way to look up information.
3 replies →
I still believe this AI push out of nowhere is due to the current Government in power state side. Making everyone question themselves and each other and being uncertain about facts while being inundated with techbro fake news called hallucinations is a recipe for disaster for older populations that don't 'trust but verify' like most technology inclined people. This is all by design and we'll falling for it.
4 replies →
I have one google account where gemini constantly confidently hallucinates crap, and another where it doesn't (at least to the extent that other modern LLMs don't).
I take this to mean that my one account has been mistaken for a competitor and they're trying to poison its data. But who knows.
Or is it randomness?
"That's the thing with randomness. You can never be sure."
Where ever would you get that impression from the company that made its fortune by being the best at searching the web?
You can tell they optimized it 100% for speed and nothing else. Web scale!
Literally every experience I've had with Gemini / Google Search AI answers followed this exact pattern, often repeated several times more if I remained persistent instead of just giving up.
Typical example:
"Where can I buy <thing I'm looking for that I can't find anywhere using normal search terms>?"
> You're looking for <related but different and widely available thing>. It is sold by <sites I never heard of>.
"No, that's different. I'm looking for <that thing but with the exact differences spelled out again>."
> Ah, my mistake. You're looking for <thing I described>. It is sold by <sites I never heard of but which don't actually sell it>.
"I've checked your links and none of those sites actually sell it, one doesn't even sell products and instead only offers manufacturing - but also not for what I asked you for."
> I'm sorry, my bad. Those sites don't sell what you are looking for. Instead you should check out <more sites I've never heard of>.
"Those sites sell the thing you initially thought I was asking about but not the thing I described."
> I'm sorry for the misunderstanding. You can find the thing you actually described at <yet more sites including some of the same>.
"No. None of these sites sell anything close to what I asked you for and two of them don't actually exist."
> Oh, sorry about that. You're completely right. The thing you asked me about isn't actually being sold by anyone. However you could buy <thing it first thought I meant and that wouldn't bring me any closer to solving my problem>.
(ad nauseam)
The few times I've resigned myself to asking Gemini or ChatGPT something I couldn't find an answer to, my experience has been the same. 100% of the time. An LLM has never, not once, given me a correct answer or not lied to me. It's always been the same experience you describe. It goes in circles "Try X... Try Y... Try X?" Until I tell it to stop telling me X or Y, and then it goes, "LOL you can't do that at all, I was just wasting your time."
Most recently, I was considering moving away from iterm2 on MacOS, and I wanted to know if any other terminal emulator supported gestures for switching between tabs. So I asked Gemini, and it says, "Yes Ghostty supports gestures for switching between tabs."
"Ok I just installed Ghostty and I can't find anything about gestures."
"You need to add foo=bar to your conf file."
"I added foo=bar to my conf file and now it's saying the conf file is invalid."
"Sorry bro, remove foo=bar and add baz=boo to the conf file."
"It says baz=boo is invalid too."
"baz=boo isn't a real option. Remove that and add foo=bar to your conf file."
"You already told me to do that and I already told you that doesn't work."
"You shouldn't put foo=bar or baz=boo in the Ghostty conf file. Both are invalid. Ghostty doesn't support gestures. Have you considered iterm2?"
Claude also has very arbitrary and confusing rules about what web pages it allows itself to look at, and how much of the page it can read. Did you know for example you’ll get a much deeper analysis if you download a PDF yourself and upload it to Claude instead of giving it the url?
This is a key reason why I actually like Grok for factual queries based on web grounding. It’s fast and reliable. Maybe it’s ignoring robots.txt? Dunno. But it works well.
Kagi works like you would expect. It searches first - and then if you’ve ended your search with a ? or configured it to always do this - it passes the search results into the assistant and gives you a summary. You can change the default model used for this if you think the cheapest, fastest model Google has is not good enough for the rare occasions you want any model’s opinions about your search results.
Google will stop being a traditional search engine because they believe they’ve found a more profitable version of it.
One where their users don’t go off to other sites and where they can keep shoving ads in their face.
The days of search for untrusted sources were numbered even without LLMs. LLMs are simply accelerating the process. Why would Google simultaneously watch one of its core products fail and fail to invest in what is likely to replace it?
I don't really buy into this theory that they want to keep all of their users on their site due to advertising revenues. The effectiveness of search engines has been degraded for decades due to SEO, and it seems as though search engines have been having an increasingly difficult time managing it in recent years. AI on the backend may help them contain it, but it comes at considerable expense. While it may help them grow their market share, it won't help them grow the market and it is a market where people expect the service for free. On the flip side, companies are already starting to sell AI services, so it can generate revenue even before advertising is factored into the picture.
I thought Google's original goal for a search engine was exactly this, answer any question whatsoever.
How did you google this, and did you use the dumb search, or specifically 'AI mode' lens?
Because I did this and got a vastly different result from you:
Yes, the Halifax Wanderers can still mathematically qualify for the 2026 Canadian Premier League (CPL) playoffs.The top four teams advance to the postseason. Following their 1-0 loss to Atlético Ottawa on September 26, 2026, the Wanderers sit in fifth place, just below the playoff line.
With a indexed table of the games and the playoff table, with a breakdown of what the points they need to achieve to do so.
The search window may not always crawl sources. AI mode specifically does some research before giving you a response. Not sure what you're on about.
> Because I did this and got a vastly different result from you
Which itself is a major UX issue. The average person is not going to understand, if they even realize, that there's a difference between the AI summary and AI mode.
One has to wonder just how much incorrect information people have consumed due to things like this.
I experienced this the other month. I searched for some safety data on something at the same time as my wife and we were effectively given extremely conflicting information about what to do by Google. Just a few tweaks in wording and different advertising profiles and Google will serve up opposite realities it seems.
Not OP, but if I’m not interested in using Google’s AI, it’s going to just return these weird bad results that it’d be better off if Google just didn’t even include them as they’re factually wrong?
Searching first is how Kagi assistant works and it's great.
I'm being very mindful of Gell-Mann Amnesia and chatbots. They speak authoritatively and are right often enough, but I've had enough cases of them saying very incorrect things in areas I know well that I have to remind myself that those cases aren't unique.
The first time, sure, it saves compute. The second time you ask, I feel like it should be the time to go into thinking/verification mode. But who are you? Are you paying? Does the answer being correct generate ad dollars? No? Then your usage mode isn't even being optimized for in their A/B test, probably.
In fact, if hallucinating the wrong answer hooks you into doing even more searches or into buying something useless, it would be preferred!
And it's going to get worse over time (kinda like how Google search results got worse over time) as ads get introduced and people try to game the ai response. General purpose AI chatbots are a waste.
> My question is: what's the point of the AI in the search engine if it itself isn't going to use the search engine first before answering?
Because the point of AI slop is to waste your time. You just lost about 30 seconds of your life trying to get a correct answer. AI was lying to you, so you had to spend time to counter the AI slop lies here.
I solved it by banning all AI slopness; in the browser some extensions do that. The world becomes better without AI slopness.