Many times you lack the context or words, but after it gave them to you, you can look up papers and figure out if it matches. Same as you do with human doctors to decide if you need a second opinion.
Yes. I guess most of the HN crowd can do this reliably, but most people cannot, and that's why doctors hate when patients google around and then lecture them based on a 5-minute WebMD skim.
Nope, nobody can really do matter expert stuff reliably like that, the HN crowd is just devs, not general geniuses, let's not sugarcoat this.
You guys know the game tekken, right? Tekken 3 and 4 were before the vast majority of llm training feeds and it shows, there's a move sequence for Bryan, taunt into while standing 1, I've dabbled in this with programmable controllers , frame by frame recording, not a single llm model could explain how and why this works.
If it would understand the game mechanics, it's trivial, but it just does not understand.
Just because a dev can read and interpret an api documentation doesn't mean he is an expert or even remotely close to that.
Arguably, llm are "good" at programming. Where's the disruption in cpp hft ?
If you have a solid self assessment of what you know, you can consider when information is supported by what you know. For the parts that aren't, you consider how critical they are, and identify the area that needs a deeper dive, where you either link it back to other areas you have confidence in your own knowledge or work to expand your own knowledge (using other resources). Then cross check knowledge. The cheap way you can do easily is to cross check between AIs/harnesses, because hallucinations happen randomly, so two different models in two different harnesses having the same hallucination is extremely rare. The more expensive option, which should be used when the information really matters, is building your own knowledge. Have the AI help guide you to resources, but not be the resource. While using non-AI resources, check you knowledge against them and the AI. Test yourself with classical resources where it really matters.
Part of it is using good judgment. Is the AI's claim reasonable if it came from another person? If some random person gave you a 2 page paper proving P=NP, would you even spend time to look at it, or would you simply toss it without caring to understand what error they made. Much like how scientists (and even the patent office) simply refuse to entertain any perpetual motion machines. Maybe if an expert in the field proposed it, and even that would depend upon how ("Hey, I found a source of free energy!" vs "Hey, I'm working through these equations and the energy term isn't balancing, can you help me find what energy source I'm missing?").
This does pose a problem. Those who are the best at understanding the limits of their knowledge and at teaching themselves need the AI's help the least but are in the position to be able to use AI with the best chance of not being misled. Those who lack these skills and would benefit the most from a personal educator, thus receving the biggest benefit in theory, are the ones most likely to fall victim to the errors made by the AI.
Consider being curious why people turn to LLMs rather than just going to authoritative sources first.
Sometimes I don't know a topic well enough to even know the right terms, or sources. But an LLM can help me become familiar with relevant concepts, and then I can more easily find the authoritative sources.
Sometimes I don't know a topic well enough to know how far it is from my understanding. An LLM helps me quickly learn how closely I have the relevant foundational knowledge: if it's absolutely gobbledygook, then I realise I've probably bitten off more than I can chew; if I half sounds familiar, then I can more confidently begin independent exploring of authoritative sources.
Ironically I wouldn't be surprised if that site will become an LLM assisted chatbot soon enough. The Dutch health care system is under pressure and GPs (and their parent organizations, and the government) are reaching towards IT automation and self-help / pre-screening / etc to reduce the load. And granted, part of the load on GPs is by people who don't need to go there, who should just go to the pharmacy or stay home for a few days. But it also feels like seeking medical help is actively being discouraged, first by the "eigen risico" (copay), now by having to jump through hoops before being able to make an appointment.
I mean, in this case it linked me to many sources, including the manual for the OCT machine used. One thing that even my ophthalmologist might not know, buried in the PDF manual, is that the sample size only includes 5 people under the age of 35 (skewing my "1st percentile" results more aggressively).
But you're right that you can't be 100% certain. Isn't that true for human-to-human explanations too?
He got a diagnosis from a real doctor. But like many specialized and highly educated people, they struggle to actually explain things clearly. Seems like Argon was able to explain it in a more understandable way.
It's not a diagnosis. It's a description of what a health problem means. If you go to the doctor empty handed and ask for explanation he'll love it even less than if you bring him nice visualization he can skim through and say, "yeah, that's about right".
Many times you lack the context or words, but after it gave them to you, you can look up papers and figure out if it matches. Same as you do with human doctors to decide if you need a second opinion.
Yes. I guess most of the HN crowd can do this reliably, but most people cannot, and that's why doctors hate when patients google around and then lecture them based on a 5-minute WebMD skim.
Nope, nobody can really do matter expert stuff reliably like that, the HN crowd is just devs, not general geniuses, let's not sugarcoat this.
You guys know the game tekken, right? Tekken 3 and 4 were before the vast majority of llm training feeds and it shows, there's a move sequence for Bryan, taunt into while standing 1, I've dabbled in this with programmable controllers , frame by frame recording, not a single llm model could explain how and why this works.
If it would understand the game mechanics, it's trivial, but it just does not understand.
Just because a dev can read and interpret an api documentation doesn't mean he is an expert or even remotely close to that.
Arguably, llm are "good" at programming. Where's the disruption in cpp hft ?
You don’t. But its shiny and pretty.
If you have a solid self assessment of what you know, you can consider when information is supported by what you know. For the parts that aren't, you consider how critical they are, and identify the area that needs a deeper dive, where you either link it back to other areas you have confidence in your own knowledge or work to expand your own knowledge (using other resources). Then cross check knowledge. The cheap way you can do easily is to cross check between AIs/harnesses, because hallucinations happen randomly, so two different models in two different harnesses having the same hallucination is extremely rare. The more expensive option, which should be used when the information really matters, is building your own knowledge. Have the AI help guide you to resources, but not be the resource. While using non-AI resources, check you knowledge against them and the AI. Test yourself with classical resources where it really matters.
Part of it is using good judgment. Is the AI's claim reasonable if it came from another person? If some random person gave you a 2 page paper proving P=NP, would you even spend time to look at it, or would you simply toss it without caring to understand what error they made. Much like how scientists (and even the patent office) simply refuse to entertain any perpetual motion machines. Maybe if an expert in the field proposed it, and even that would depend upon how ("Hey, I found a source of free energy!" vs "Hey, I'm working through these equations and the energy term isn't balancing, can you help me find what energy source I'm missing?").
This does pose a problem. Those who are the best at understanding the limits of their knowledge and at teaching themselves need the AI's help the least but are in the position to be able to use AI with the best chance of not being misled. Those who lack these skills and would benefit the most from a personal educator, thus receving the biggest benefit in theory, are the ones most likely to fall victim to the errors made by the AI.
Sometimes you can go to the websites of your country's health care system, look up what it said, read and compare.
I'm an American and I might trust the chat bot more on some topics right now.
Would be nice if people just did that instead.
It's kind of infuriating to hear Dutch people cite these chatbots without bothering to check https://www.thuisarts.nl/.
Consider being curious why people turn to LLMs rather than just going to authoritative sources first.
Sometimes I don't know a topic well enough to even know the right terms, or sources. But an LLM can help me become familiar with relevant concepts, and then I can more easily find the authoritative sources.
Sometimes I don't know a topic well enough to know how far it is from my understanding. An LLM helps me quickly learn how closely I have the relevant foundational knowledge: if it's absolutely gobbledygook, then I realise I've probably bitten off more than I can chew; if I half sounds familiar, then I can more confidently begin independent exploring of authoritative sources.
3 replies →
Ironically I wouldn't be surprised if that site will become an LLM assisted chatbot soon enough. The Dutch health care system is under pressure and GPs (and their parent organizations, and the government) are reaching towards IT automation and self-help / pre-screening / etc to reduce the load. And granted, part of the load on GPs is by people who don't need to go there, who should just go to the pharmacy or stay home for a few days. But it also feels like seeking medical help is actively being discouraged, first by the "eigen risico" (copay), now by having to jump through hoops before being able to make an appointment.
1 reply →
I mean, in this case it linked me to many sources, including the manual for the OCT machine used. One thing that even my ophthalmologist might not know, buried in the PDF manual, is that the sample size only includes 5 people under the age of 35 (skewing my "1st percentile" results more aggressively).
But you're right that you can't be 100% certain. Isn't that true for human-to-human explanations too?
It's easier to sense how confident someone is in their knowledge
But the correlation between confidence and correctness, especially between different speakers, is famously missing.
How do you know that for a web site? You have to research and cross check, and talk to a real doctor.
Easy. Just feed the results to a second frontier model to verify. /s
Ask a doctor?
+1. I've had successes from LLM-as-physio, but continuing to see the physio to make sure I'm not going mad.
"Just see a physio" misses that:
* I want the physio's advice during workouts
* I want the physio's advice during moments of pain
* I want the physio to explain things in more detail, but my 30-60 minute appt is not enough, and it's difficult/expensive to book them for longer.
Right, doctors love when patients bring their ChatGPT generated diagnoses for them to read over, it saves them so much work.
He got a diagnosis from a real doctor. But like many specialized and highly educated people, they struggle to actually explain things clearly. Seems like Argon was able to explain it in a more understandable way.
It's not a diagnosis. It's a description of what a health problem means. If you go to the doctor empty handed and ask for explanation he'll love it even less than if you bring him nice visualization he can skim through and say, "yeah, that's about right".