Comment by bwb

Can you explain the exact way in which this is possible? It’s not legal to be denied jobs based on health. Not to deny insurance

These strawman arguments lack nuance.

If the person can use AI to lead a noticeably better life, something that may have been impossible previously due to economic circumstance, then the first order benefits outweigh the second order drawbacks.

I’m not disputing what you’re saying, I just think that treating it like a zero sum game every time the conversation comes up is showing an immense amount of privilege.

You, me, the parent commenter; we’re all dying, we don’t have time to optimise for the best outcome.

> when we could have had an open world with open models run locally instead where you got to keep your private health information private

But we can have that? If you have powerful enough hardware you can do it, right now. At very least until the anti-AI people get their way and either make the models' creators liable for what the models say or get rid of the "training is fair use" thing everyone depends on, in which case, sure, you'll have to kiss legal open-weight models goodbye.

Precisely right. Related. https://www.socialcooling.com/

This is an argument against the general data collection internet NOT chatGPT.

What do you consider the purpose of life to be? To me being in good health is immensely more important than health insurance, a government health plan, or a job.

I know that neither health insurers nor any government agency nor anybody else have even 0,0000000000000001% as much interest in my health, well being and survival as I do.

When it is the matter of my health and my life, I care as much about what an insurer or employer thinks as I would care about what the Ayatollah of Iran thinks. Or what you think. Ie: Those opinions are without any value at all.

System working as intended!

If an insurer is able to reduce (or recoup) costs from likelier risks, then the remaining insureds benefit from lower premiums.

If the goal is providing subsidies (i.e. wealth transfers), then insurance is not the way to do it. That is the government’s role.

This is the crux of the argument from the article.

> get to know your members even before the first claim

Basically selling your data to maximise profits from you and ensure companies don't take on a burden.

You are also not protected by HIPAA using ChatGPT.

Because we've all used LLMs.

The make stuff up. Doctors do not make stuff up.

They agree with you. Almost all the time. If you ask an AI whether you have in fact been infected by a werewolf bite, they're going to try and find a way to say yes.

You could also list plenty of horror stories where people went to medical professionals and got screwed over. There is this myth that people can go to doctors and get perfect attention and treatment. Reality is far from that

"…a 60-year-old man who had a “history of studying nutrition in college” decided to try a health experiment: He would eliminate all chlorine from his diet…"

You can see already that this can easily go sideways. This guy is already exploring the nether regions of self-medication.

It would be ideal if LLMs recognized this and would not happily offer up bromine as a substitute for chlorine, but I suspect this guy would have greedily looked for other shady advice if LLMs had never existed.

No, there's a difference between radically changing your diet and changing up your stretch/strength routine.. you don't just "end up" like one of them, you can evaluate that the downside risk of the latter is much lower and try it safely while recognizing that an extreme diet might not be so safe to try without any professional guidance.

You have to use your head, just like online forums or with doctors :)

I've had doctors tell me to do insane things. Some that caused lasting damage. Better to come with a trust-but-verify attitude to humans and AI.

Natural selection at work. I don’t see anything suspicious here.

The man in the article did not use it as a research help and did not verify it with experts.

So what's your argument?

Did he also drive into a lake following Google Maps' driving directions?

I agree with that. The question I suppose is whether an LLM can detect, perhaps by the question itself, if they are dealing with someone (I hate to say it) "stable".

Anyone asking how to commit suicide, as a recent example, should be an obvious red flag. We can get more nuanced from there.

That was a specific example where AI helped me revamp my workout. No workout stays static; it has evolved over the years, in small ways. With this change I threw out 60% of it, and replaced it with a lot of work to make sure I was working full body and taking into account the injury. I was having more pain in 2025 and was only pain-free around 90% of the time. I wanted to get stronger but also revet everything I was doing to hopefully get out of that pain.

Hope that helps!

You can ask a trainer questions, they are super helpful and taught a lot, but it is still one person, and they don't often sit around at night reading university/research papers on the injury, etc.

> And if it didn't work out and made you worse or, god forbid, the advice caused you to get seriously injured, then what? ChatGPT won't take any responsibility.

Realistically in 99% of actual cases where this happens due to human medical advice, the humans too won't take any responsibility.

You always have to use critical thinking, listen to your body, and get advice from trainers in the trenches. As I mentioned, I did all of those things :)

on a similar vein, I have recurring back issues due to a spinal issue. I gave the issue to ChatGpT and it gave me almost all of the exercise I had been given years ago by a chiropractor. It's nowhere near a replacement for having someone coach me through movements though.

Totally agree, it can be a bit of an echo chamber. I had an infection post-dental-work. Bing Chat insisted I had swollen lymph nodes from a cold that would resolve on their own, then decided I had a salivary gland infection. After a follow-up with a real-world ENT, it was (probably accurately) diagnosed as a soft-tissue infection that had completely resolved on two rounds of antibiotics. The AI never raised that possibility, whereas the ENT and dentist examined me and reached that conclusion immediately.

I do think AI is great for discussing some health things (like "how should I interpret this report or test result?"), but it's too echo chamber-y and suggestion-prone for accurate diagnosis right now.

We have to also understand that the trainer didn't get to spend 12 hours of researching every minutia or do a trial and error study to get to where OP got to. This doesn't necessarily mean the trainer failed, just that they were constrained by time, which OP wasn't. And I think that is the essence of this tech, when used wisely, I can lead to results like these which you can't get despite having access to the best talent for a limited time. Only the well afforded can afford a full time trainer/therapist.

The trainer was a godsend, got me to ~95% pain-free, and taught me all kinds of things. He is amazing.

But 2025 was maybe down to 90% pain-free, and I want to get stronger. So I did a big rewrite of my entire workout plan and checked everything. AI wasn't perfect, but it was amazing when you already know some.

It is still a tool I had to direct, and it took a few days of work. But I'm amazed at where it got me to. It took the injury into consideration and my main sport, and built around that. In the past I tried do this online and couldn't do it given the numerous factors involved. It was not perfect, but over the course of a few days, I was able to sort it out (and test with a trainer on the approach a few weeks after).

I've been 100% pain free for 6 weeks in a way I haven't felt in a long time.

Like anything, it is a tool; someone using WebMD badly, and someone can use it well.

I have found it helpful as I can ask ChatGPT questions, teach myself about what I am dealing with, and understand it better so I can ask my doctor questions. I still verify a lot, I still read articles on verified medical sites, etc., but it helps me do that a lot quicker, and I seem to learn quicker.

I'm sure someone can also go deep into anxiety with it as well if they approach it that way. It isn't a miracle button, but it is an AMAZING tool IME.

One of the big issues I have with LLMs that when you start a prompting session with an easy question it all goes great. It bring up points you might not have considered and appears very knowledgeable. Fact checking at this stage will show the LLM is invariably correct.

Then you start "digging deeper" on a specific sub-topic, and this is where the risk of an incorrect response grows. But it is easy to continue with the assumption the text you are getting is accurate.

This has happened so many times with the computing/programming related topics i usually prompt about, there is no way I would trust a response from an LLM on health related issues I am not already very familiar with.

Given that the LLM will give incorrect information (after lulling people with a false sense of it being accurate), who is going to be responsible for the person that makes themselves worse off by doing self diagnosis, even with a privacy focused service?

I'm lucky to live in Europe now, I cried the first time I went to a doctor here, he chatted with me for 45 minutes. I begged my doctor in the USA to let me book back-to-back sessions, so I could ask him questions and better understand what was going on. He said no; I only had 10 minutes, and he generally didn't have time to answer any questions or provide details. He was a good doc, but just couldn't take the time, and insurance wouldn't comp him for back to back appointments.

You have to use critical thinking + it helps to have some info on the subject + it shouldn't be used to perform self-surgery :)

I spent about 12 hours over 2 days, checking, rechecking, and building out a plan. Then I did 2-hour sessions on YouTube, over several weeks, learning the new exercises with proper form (and that continues as form is hard). Followed by an appointment with a trainer to test my form and review the workout as a hole (which he approved of). No trainer really knows how this injury will manifest, so a lot is also helped because I have 10 years of exp.

This isn't a button click, and now follow the LLM lemming. This is a tool like Google search but better.

I could not have done this before using the web. I would have had to read books and research papers, then try to understand which exercises didn't target x muscle groups heavily, etc. I just couldn't do that. The best case would have been a trainer with the same injury, maybe.

You are exaggerating. LLMs simply don’t hallucinate all that often, especially ChatGPT.

I really hate comments such as yours because anyone who has used ChatGPT in these contexts would know that it is pretty accurate and safe. People also can generally be trusted to identify good from bad advice. They are smart like that.

We should be encouraging thoughtful ChatGPT use instead of showing fake concern at each opportunity.

Your comment and many others just try to signal pessimism as a virtue and has very less bearing on reality.

I have this same reaction.

But I also have to honestly ask myself “aren’t humans also prone to make stuff up” when they feel they need to have an answer, but don’t really?

And yet despite admitting that humans hallucinate and make failures too, I remain uncomfortable with ultimate trust in LLMs.

Perhaps, while LLMs simulate authority well, there is an uncanny valley effect in trusting them, because some of the other aspect of interacting with an authority person are “off”.

It isn't a 0 or a 1; it is a spectrum. Doctors don't understand everything, either, which is the scary thing we don't like to realize.

What it specifically helps me to understand are things like: probable outcomes, symptoms in greater detail, as well as how they manifest in patient populations, explains it as if I was a 10/15/20 year old in detail to help me understand the basics of what might be going on, similar things in possible pharma options, general response treatments and pros/cons, etc.

I'm not using this to perform self-surgery or build a belief system :), I'm just trying to learn and understand what is going on at a better level.

Hope that helps :)

It's a lot easier to double-check, cross-reference, or test the validity of advice given by a gpt. It has no authority, no persuasion mechanisms, and its opinions are there in plain text ready to be picked apart. You can ask it for references and non-confrontationally challenge it on the things you're sceptical about. It generally avoids woo in my experience though it's hardly always correct in specific advice, it can definitely point you in productive directions. Which is completely different from discussing anything health with a homeopath, who at best will try to get you to avoid productive treatment and at worst poison you.

> You are in no position to discern the validity of your "knowledge".

He is the only one who is in that position, because he is the only person who is inside his body. He is physically and mentally a hundred percent in the position to discern the validity of the advice.

I've been on the web since it was born. When will this happen? :)

How is this manifesting in reality?

You bring up an interesting point.

How is it we have come to a place in society where we second-guess everything we type? But perhaps also where we go (with our "tracking devices" in our pockets…).

I mean, obviously the internet is what changed everything. But it is like you have a megaphone strapped to your face whenever you connect to a site and make a comment.

Maybe this is not a good thing.