Comment by transcriptase
14 days ago
I have to wonder how many CEOs and other executives are low-key bouncing their bad ideas off of ChatGPT, not realizing it’s only going to tell them what they want to hear and not give genuine critical feedback.
14 days ago
I have to wonder how many CEOs and other executives are low-key bouncing their bad ideas off of ChatGPT, not realizing it’s only going to tell them what they want to hear and not give genuine critical feedback.
"Low-key?" About that. https://futurism.com/openai-investor-chatgpt-mental-health
> As such, if he really is suffering a mental health crisis related to his use of OpenAI's product, his situation could serve as an immense optical problem for the company, which has so far downplayed concerns about the mental health of its users.
Yikes. Not just an optics* problem, but one has to consider if they're pouring so much money into the company because he feels he "needs" to (whatever basis of coercion exists to support his need to get to the "truth").
That's bizarre. I wonder if the use of AI was actually a contributing factor to his psychotic break as the article implies, or if the guy was already developing schizophrenia and the chat bot just controlled what direction he went after that. I'm vaguely reminded of people getting sucked down conspiracy theory rabbit holes, though this seems way more extreme in how unhinged it is.
In form, the conversation he had (which appears to have ended five days ago along with all other public footprint) appears to me very much like a heavily refined and customizable version of "Qanon," [1] complete with intermittent reinforcement. That conspiracy theory was structurally novel in its "growth hacking" style of rollout, where ARG and influencer techniques were leveraged to build interest and develop a narrative in conjunction with the audience. That stuff was incredibly compelling when the Lost producers did it in 2010, and it worked just as well a decade later.
Of course, in 2020, it required people behind the scenes doing the work to produce the "drops." Now any LLM can be convinced with a bit of effort to participate in a "role-playing game" of this type with its user, and since Qanon itself was heavily covered and its subject matter broadly archived, even the actual structure is available as a reference.
I think it would probably be pretty easy to get an arbitrary model to start spitting out stuff like this, especially if you conditioned the initial context carefully to work around whatever after-the-fact safety measures may be in place, or just use one of the models that's been modified or finetuned to "decensor" it. There are collections of "jailbreak" prompts that go around, and I would expect Mr. Jawline Fillers here to be in social circles where that stuff would be pretty easy to come by.
For it to become self-reinforcing doesn't seem too difficult to mentally model from there, and I don't think pre-existing organic disorder is really required. How would anyone handle a machine that specializes in telling them exactly what they want to hear, and never ever gets tired of doing so?
Elsewhere in this thread, I proposed a somewhat sanguine mental model for LLMs. Here's another, much less gory, and with which I think people probably are a lot more intuitively familiar: https://harrypotter.fandom.com/wiki/Mirror_of_Erised
[1] https://en.wikipedia.org/wiki/QAnon#Origin_and_spread
2 replies →
Is "futurism.com" a trustworthy publication? I've never heard of it. I read the article and it didn't seem like the writing had the hallmarks of top-tier journalism.
I'm not familiar with the publication either, but the claims I've examined, most notably those relevant to the subject's presently public X.com The Everything App account, appear to check out, as does that the account appears to have been inactive since the day before the linked article was published last week. It isn't clear to me where the reputation of the source becomes relevant.
Fun fact, my manager wrote my annual review with our internal LLM tool, which itself is just a wrapper around GTP-4o.
(My manager told me when I asked him)
This already seems somewhat widespread. I have a friend working at a mid-tier tech co that has a handful of direct reports. He showed me that the interface to his eval app had a “generate review” button, which he clicked, then moved on to the next one.
Honestly, I’m fine with this as long as I also get a “generate self review” button. I just wish I could get back all the time I’ve spent massaging a small number of data points into pages of prose.
That makes you wonder why we go through the ritual of an annual review if no one takes it seriously.
I think I have a precise figure : it's a lot.
Yeah. If HN is your primary source of in depth AI discussion, you get a pretty balanced take IMO compared to other channels out there. We (the HN crowd) should take into account that if you take "people commenting on HN" as a group, you are implicitly selecting for people that are able to read, parse and contextualise written comment threads.
This is NOT your average mid-to-high level corpo management exec, who can for more than 80% (from experience) be placed in the "rise of the business idiot" cohort, fed on prime linkedin brainrot. self-reinforcing hopium addicts with an mba.
Nor is it the great masses of random earth dwellers who are not always able to resist excess sugar, nicotine, mcdonalds, youtube, fentanyl, my-car-is-bigger-than-yours credit card capitalism, free pornography, you name it. And now RLHF: Validation as a service. Not sure if humanity is ready for this.
(Disclosure: my mum has a chatgpt instance that she named and I'm deeply concerned about the spiritual convos she has with it; random people keep calling me on the level of "can you build me an app that uses llms to predict Funko Pop futures".)
And we thought having access to someones internet searches was good intel. Now we have a direct feed to their brain stem along with a way to manipulate it. Good thing that narcissistic sociopaths have such a low expression in the overall population.
[flagged]
Then it'll be no different than when they bounce their bad ideas off their human subordinates.
This. Their immediate subordinates will be just a as sycophantic, if not more.
cf Celene's Second Law.
I feel like I’ve had good results for getting feedback on technical writing by claiming the author is a third party and I need to understand the strengths and weaknesses of their work. I should probably formally test this.
It depends what model you use. o3 pushes back reasonably well. 4o doesn't.
TBH if they have sycophants and a lot of money it's probably the same. How many bullshit startups have there been, how many dumb ideas came from higher up before LLMs?
[flagged]