Comment by joegibbs

Most non-technical people have an inaccurate mental model of how LLMs work, based mostly on sci-fi movies like 2001 or Terminator, where they anthromorphise the model and assume that since it gives more accurate responses to most questions than any human they know that it must be some kind of superhuman genius master computer, sitting in the OpenAI headquarters pondering deep questions about reality.

Of course, when you go and finetune a model yourself, or run a smaller model locally, it becomes obvious that it doesn't have any mystical abilities. But almost nobody does this. I'm no anti-AI crusader, I use them all the time and think they're great and there's plenty of potential to use them to grow the economy, but the hype is insane.

It doesn't help when influencers like Yudkowsky go on anthromorphising about what it "knows", or go on with that mystical nonsense about shoggoths, or treat dimensionality of embeddings as if the model is reaching into some eldritch realm to bring back hidden knowledge, or hype up the next model talking about human extinction or anything like that.

It also doesn't help when the companies making these models:

- Use chat interfaces where the model refers to itself as "I" and acts as if it's a person

- Tune them to act like your best buddy and agree with you constantly

- Prompt the LLM with instructions about being an advanced AI system, pushing it toward a HAL or Skynet-type persona

- Talk about just how dangerous this latest model is

You tell these things that they're a superhuman AI then of course they're going to adopt all the sci-fi tropes of a superhuman AI and roleplay that. You tell them they're Dan the bricklayer they're going to act as if they're Dan the bricklayer.