Comment by wren6991
6 hours ago
Speaking to the "uncensored model" angle: there's little reason to distribute abliterated weights anyway. Instead of orthogonalising the weights that write back to the residual stream, you can just orthogonalise the activations themselves. It's equivalent.
Orthogonalising activations at runtime is computationally cheap. Just distribute the refusal vectors (few thousand floats per layer), then run against the stock weights. Antirez's DS4 already supports this: https://github.com/antirez/ds4/blob/8db1d1d155cb0400a86a86b9...
Abliterated weights are just a bad habit we've gotten into. It's also deeply suboptimal from a precision point of view to take a model that's already been QATed and distributed in pre-quantised form (DeepSeek V4, Kimi K2.5 or K3...), modify its weights, and re-quantise it. Similarly, abliterated models regain some of their refusal behaviour when they're re-quantised after abliteration -- avoidable by keeping the two separate.
A question that comes up in my mind, since I don't fully understand how this works, is how does this affect runtime performance. It feels like abliterated weight models would work faster than some extra runtime operations?
I believe abliterated models are mostly still created at this point because they're "universal": they can not only be run locally, and on cloud GPUs, but also on "managed inference" providers (i.e. services where you hand them a model URI, and they blindly fetch it, load it, and give you inference access to it through standard text/chat-completion APIs. Think HuggingFace Spaces, or Google CoLab, or CloudFlare Workers AI.)
Such managed inference providers have (for now) plausible deniability of behaving ethically (at least enough that they don't get boycotted / scare away investors) due to them being "blind" to what gets run on their systems. They're acting as the inference equivalent of data transit carriers.
But I don't think it would be possible for managed inference providers to publicly expose "runtime activation steering" in the way antirez's DS4 does, without that reading much more explicitly as them inviting unethical workloads.
(Yes, there are other things you can do with runtime steering. But almost all of those things are workload-specific, relying on you privately tuning to the needs of your own dataset. And if you can do that, you can run inference without the help of a managed inference provider. The only time a customer will come along with a pre-made runtime-steering vector file in hand, is if that vector is an alignment-orthogonalization vector.)
Yeah I use a custom fork of llama.cpp that has an abliteration feature that basically does this. It's sloppily vibe coded and I don't have time to coordinate on a way to do this cleanly upstream, but it's absolutely possible and saves a lot of time and bandwidth from being wasted
Interesting. Is there a paper that explains this in a bit more detail, like [0] for abliteration (underlying the Heretic software, afaik)?
[0] https://arxiv.org/abs/2406.11717
That paper you linked has all of the information you need. The linear algebra they do on the weights there to null out one direction reduces to a single (dot-product + broadcast-multiply + add) on the activations.
Can you explain this a bit to a non-expert?
I haven't wrapped my mind around this
This is the original description of abliteration and it's quite approachable and interesting to read: Refusal in Language Models Is Mediated by a Single Direction (https://arxiv.org/abs/2406.11717). Warning: changes to your world view caused by seeing "HarmBench" used to maximise expected harm instead of minimising it may be irreversible.
There's an empirical observation that models often have a single direction in their activation space for "hmm no I shouldn't do this". It forms naturally during pre-training, and is then surfaced during post-training to make the model refuse to engage in certain behaviour.
With a little bit of linear algebra you can zap that direction from the model's activations, and it stops refusing to do things. You can also do the opposite: magnify that direction, and the model refuses to do anything at all.
I'm pretty sure this was achieved with prompting rather than with weights, but there is a chatbot available that tries to maximize the motivated refusals:
https://www.goody2.ai/chat
Damn what's happened since this? Presumably they scramble refusal intentionally somehow now? Like intentionally couple it to "directions" that effect performance if messed with? Or is it more like just don't rely on the model to refuse and instead capture bad responses between generation and delivery?
2 replies →
Instead of editing the weights so they don't create the refusal signal, just let them do whatever, then delete the refusal signal itself. You don't want to edit quantised weights because it causes a loss of precision that can be pretty bad.
I didn’t know about that method, thank you. I’ve needed a local model for security research but Qwen 27b abliterated did 30% worse than the stock weights on my internal benchmarks (I just skip the public benches now, it’s honestly useless noise on an operational level).
So... distribute a LoRA (or equivalent) that modifies the base weights with the abliteration vectors. That makes sense as it would be possible to try different abliterations and keep the storage space down.
Yeah, orthogonalisation is a special case of a rank-1 LoRA.
I don’t think this is true.
1 reply →
Who is offering the refusal vectors or instructions on how to generate them?
Here's a widely used toolkit: https://github.com/p-e-w/heretic
Distributing the vectors themselves isn't (yet) common practice, because people have gotten used to just putting the full modified weights up on HuggingFace's huge free storage.
I am, in fact, I’m offering something much better, a compact format that exactly restores an abliterated model from a tiny JSON file:
https://www.reddit.com/r/LocalLLaMA/s/iksvmXBzuC
> Similarly, abliterated models regain some of their refusal behaviour when they're re-quantised after abliteration
Thanks for this information, Q4 seemed fine but they reappeared again in Q5 with an vengeance, I couldn't understand why. Very Strict and I've only found one jail break that barely works around 60% of the time.