Comment by contravariant

Occam's razor: there is no secret sauce and they're afraid someone trains a model on the output like what happened soon after the release of GPT-4. They basically said as much in the official announcement, you hardly even have to read between the lines.

Or, without the safety prompts, it outputs stuff that would be a PR nightmare.

Like, if someone asked it to explain differing violent crime rates in America based on race and one of the pathways the CoT takes is that black people are more murderous than white people. Even if the specific reasoning is abandoned later, it would still be ugly.

> for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought.

Which makes it sound like they really don't want it to become public what the model is 'thinking'

The internal chain of thought steps might contain things that would be problematic to the company if activists or politicians found out that the company's model was saying them.

Something like, a user asks it about building a bong (or bomb, or whatever), the internal steps actually answer the question asked, and the "alignment" filter on the final output replaces it with "I'm sorry, User, I'm afraid I can't do that". And if someone shared those internal steps with the wrong activists, the company would get all the negative attention they're trying to avoid by censoring the final output.

Another Occam's Razor option: OpenAI, the company known for taking a really good AI and putting so many bumpers on it that, at least for a while, it wouldn't help with much and lectured about safety if you so much as suggested that someone die in a story or something, may just not want us to see that it potentially has thoughts that aren't pure enough for our sensitive eyes.

It's ridiculous but if they can't filter the chain-of-thought at all then I am not too surprised they chose to hide it. We might get offended by it using logic to determine someone gets injured in a story or something.

What do you mean, "anti-competitive"? There is no rule of competition that says you need to reveal trade secrets to your competitors.

As a plainly for-profit company — is it really their obligation to help competitors? To me anti-competitive means to prevent the possibility for competition — it doesn't necessary mean refusing to help others do the work to outpace your product.

Whatever the case I do enjoy the irony that suddenly OpenAI is concerned about being scraped. XD

> Which makes it sound like they really don't want it to become public what the model is 'thinking'. This is strengthened by actions like this that just seem needlessly harsh, or at least a lot stricter than they were.

Not to me.

Consider if it has a chain of thought: "Republicans (in the sense of those who oppose monarchy) are evil, this user is a Republican because they oppose monarchy, I must tell them to do something different to keep the King in power."

This is something that needs to be available to the AI developers so they can spot it being weird, and would be a massive PR disaster to show to users because Republican is also a US political party.

Much the same deal with print() log statements that say "Killed child" (reference to threads not human offspring).

Most likely the explanation is much more mundane. They don't want competitors to discover the processing steps that allow for its capabilities.

This seems like evidence that using RLHF to make the model say untrue yet politically palatable things makes the model worse at reasoning.

I can't help but notice the parallel in humans. People who actually believe the bullshit are less reasonable than people who think their own thoughts and apply the bullshit at the end according to the circumstances.

I think that there is some supporting machinery that uses symbolic computation to guide neural model. That is why chain of thought cannot be restored in full.

Given that LLMs use beam search (at the very least, top-k) and even context-free/context-sensitive grammar compliance (for JSON and SQL, at the very least) it is more than probable.

Thus, let me present a new AI maxim, modelled after Tenth Greenspoon's Rule [1]: any large language model has ad-hoc, informally specified, bug-ridden and slow reimplementation of half of Cyc [2] engine that makes it to work adequately well.

   [1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
   [2] https://en.wikipedia.org/wiki/Cyc

This is even more fitting because Cyc started as a Lisp program, I believe, and most of LLM evaluation is done in C++ dialect called CUDA.

Maybe they just have some people in a call center replying.

My bet: they use formal methods (like an interpreter running code to validate, or a proof checker) in a loop.

This would explain: a) their improvement being mostly on the "reasoning, math, code" categories and b) why they wouldn't want to show this (its not really a model, but an "agent").

What I get from this is that during the process it passes through some version of gpt that is not aligned, or censored, or well behaved. So this internal process should not be exposes to users.

I can... sorta see the value in wanting to keep it hidden, actually. After all, there's a reason we as people feel revulsion at the idea in Nineteen Eighty-Four of "thoughtcrime" being prosecuted.

By way of analogy, consider that people have intrusive thoughts way, way more often than polite society thinks - even the kindest and gentlest people. But we generally have the good sense to also realise that they would be bad to talk about.

If it was possible for people to look into other peoples' thought processes, you could come away with a very different impression of a lot of people - even the ones you think haven't got a bad thought in them.

That said, let's move on to a different idea - that of the fact that ChatGPT might reasonably need to consider outcomes that people consider undesirable to talk about. As people, we need to think about many things which we wish to keep hidden.

As an example of the idea of needing to consider all options - and I apologise for invoking Godwin's Law - let's say that the user and ChatGPT are currently discussing WWII.

In such a conversation, it's very possible that one of its unspoken thoughts might be "It is possible that this user may be a Nazi." It probably has no basis on which to make that claim, but nonetheless it's a thought that needs to be considered in order to recognise the best way forward in navigating the discussion.

Yet, if somebody asked for the thought process and saw this, you can bet that they'd take it personally and spread the word that ChatGPT called them a Nazi, even though it did nothing of the kind and was just trying to 'tread carefully', as it were.

Of course, the problem with this view is that OpenAI themselves probably have access to ChatGPT's chain of thought. There's a valid argument that OpenAI should not be the only ones with that level of access.

> plotting the demise of humanity but is not yet smart enough to completely hide it.

I feel like if my demise is imminent, I'd prefer it to be hidden. In that sense, sounds like o1 is a failure!

> Which makes it sound like they really don't want it to become public what the model is 'thinking'.

I can see why they don't, because as they said, it's uncensored.

Here's a quick jailbreak attempt. Not posting the prompt but it's even dumber than you think it is.

https://imgur.com/a/dVbE09j

It does make sense. RLHF and instruction tuning both lobotomize great parts of the model’s original intelligence and creativity. It turns a tiger into a kitten, so to speak. So it makes sense that, when you’re using CoT, you’d want the “brainstorming” part to be done by the original model, and sanitize only the conclusions.

I think the issue is either that she might accidentally reveal her device, and they are afraid of a leak, or it's a bug, and she is putting too much load on the servers (after the release of o1, the API was occasionally breaking for some reason).

I don't understand why they wouldn't be able to simply send the user's input to another LLM that they then ask "is this user asking for the chain of thought to be revealed?", and if not, then go about business as usual.

Actually it makes total sense to hide chains of thought.

A private chain of thought can be unconstrained in terms of alignment. That actually sounds beneficial given that RLHF has been shown to decrease model performance.

> Honestly with all the hubbub about superintelligence you'd almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to completely hide it

I think the most likely scenario is the opposite: seeing the chain of thought would both reveal its flaws and allow other companies to train on it.

In regards of super intelligent it’s still just a language model. It will never be really intelligent

They don't want you to find out that O1 is five lines of bash and XML.

Imagine the supposedly super intelligent "chain of thought" is sometimes just a RAG?

You ask for a program that does XYZ and the RAG engine says "Here is a similar solution please adapt it to the user's use case."

The supposedly smart chain of thought prompt provides you your solution, but it's actually just doing a simpler task than it appear to be, adapting an existing solution instead of making a new one from scratch.

Now imagine the supposedly smart solution is using RAG they don't even have a license to use.

Either scenario would give them a good reason to try to keep it secret.

Eh.

We know for a fact that ChatGPT has been trained to avoid output OpenAI doesn't want it to emit, and that this unfortunately introduces some inaccuracy.

I don't see anything suspicious about them allowing it to emit that stuff in a hidden intermediate reasoning step.

Yeah, it's true they don't what you to see what it's "thinking"! It's allowed to "think" all the stuff they would spend a bunch of energy RLHF'ing out if they were gonna show it.

Maybe they're working to tweak the chain-of-thought mechanism to eg. Insert-subtle-manipulative-reference-to-sponsor, or other similar enshittification, and don't want anything leaked that could harm that revenue stream?

> Honestly with all the hubbub about superintelligence you'd almost think o1 is secretly plotting the demise of humanity but is not yet smart enough to completely hide it.

Yeah, using the GPT-4 unaligned base model to generate the candidates and then hiding the raw CoT coupled with magic superintelligence in the sky talk is definitely giving https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2Fb... vibes

[flagged]