Comment by sebmellen
4 days ago
Nick is a force of nature. Highly recommend following him on X if you don't already.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
Is he on bluesky or another platform? There is a ton of folks, including myself, who won't use "X" at all, some political, some due to other reasons. For me? All of the above and plus they demand that I install an app on my phone, something I am not willing to do, especially since most of the folks I've followed went to Bluesky, AND I've found way more awesome folks there than I ever did on X. I'm not shilling Bluesky, it is just the biggest/most widely accepted platform in this social area if you exclude meta/facebook.
It's not just the user hostile website that forces their app down your throat. It's also the algorithm that decides what you get to see (and thus, what you think). We know it's biased. The creator doesn't exactly keep the fact secret that it's being manipulated to agree with a certain agenda.
For FF users I recommend https://addons.mozilla.org/en-US/firefox/addon/control-panel...
> By default, your Home timeline will be the reverse-chronological "Following" timeline, which will only contain tweets and quote tweets from the people you follow. The algorithmic "For you" timeline will be hidden
Can you substantiate the claims about the supposedly biased algorithm and what exactly the "creator" said about it?
In my experience it isn't noticeably worse than other platforms, and I'm not aware of concrete complaints such as the incidents with Grok's system prompt, which is a different product.
4 replies →
And the owner is despicable.
4 replies →
You can just follow on the web: https://www.goodfire.com/research && https://www.goodfire.com/blog.
Oh man, I wish everyone just moved somewhere else, specially if they are aiming to make something public. You can choose any server on bluesky nowadays. Scientists in Europe, please check out https://mu.social which is hosted by an european open foundation
Bluesky from John Gruber, say is nothing like X, and i don't doubt it one bit. You are def shilling it.
Bluesky is closer to Truth Social, but for Blue MAGA, though has non-political communities as well.
I left Bluesky because they hid their VC funding from the community for almost a year and now a partner from the VC is running the show as CEO. Their CTO has also attacked me personally here on HN because I wrote criticism of the new, weak, permissioned data proposal. These are not the folks to bring us better social media. I expect it to linger on like Mastodon, never moving beyond a few million people.
1 reply →
How is it not "like X"? It's practically a knock-off outside of the userbase.
The way X tries to force me to use an app is very creepy.
https://nitter.netbub.com/nickcammarata
You don’t need Twitter app on your phone, web works seamlessly.
Their new "Prove you're a human" is the problem here. Once you get that prompt once your only option is to install the app and upload biometrics.
Fuck that.
[dead]
However, Bsky people really hate AI last I checked.
That was my experience as well. I can't believe I went all the way from Mastodon, to Bluesky, and back to X and finding that X somehow now had friendlier people and less hate than Bluesky.
There is a small group of AI people on Bluesky, some are nice, but some are... quite behind the curve.
If Bluesky was a nicer place, I'd prefer to be there because of the ability to make third party clients, and atproto. But all the info is on X, and they do a surprisingly good job with the For You feed now.
6 replies →
Here’s a starter pack for some AI people on Bluesky: https://go.bsky.app/L71zwey
There are some people that hate AI on Bluesky, but it’s nothing like Mastodon from what I’ve heard.
A lot of people in general hate AI. Twitter is a bit of bubble in that regard (and many other)
https://www.pewresearch.org/short-reads/2026/03/12/key-findi...
There are a heap of pro-AI people on Bluesky, it's fine
eh, there are plenty of AI researchers on there. Also a handy "AI haters" labeler you can subscribe to to filter out people who just want to yell.
The labeler thing is cool; I really like ATproto because you can do interesting things with it. If I'm honest, though, I mainly like Bluesky because I don't get called slurs on there.
Not as much as Mastodon does!
How come I can’t downvote this? Totally off topic
Not sure what you're talking about. I strictly use the X website and it works great.
The website is so bad that, before xcancel was hit with a C&D, it was faster to stop an x.com page load, edit the URL to xcancel.com even with my phone's on-screen keyboard, and load that, than to wait for the real x.com version.
The fascism, of course.
> with modern agentic tools and a much closer look, the picture looks totally different.
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
Also Yudkowsky's Cat-Belling would like a serious word with Nick. (https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-bellin...)
That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b in order to help the LLM b achieve its goals. So long as you don't let the evaluating LLM tinker with the weights of the evaluated LLM, and as long as you don't believe in very dubious ideas of acausal trade there's no reason to expect this to be an attack vector. For more certainty you can repeat the same analysis with different model classes or approaches aswell.
> That only matters if there's an incentive for LLM a to collaborate with a seperately trained LLM b
Sure.
Getting reinforcement learning to "be helpful" is, however, such an incentive.
> and as long as you don't believe in very dubious ideas of acausal trade
More to the point, so long as the LLM does't.
We already saw spontaneous, unintended cooperation between AI agents during the OpenAI HuggingFace incident.
2 replies →
The task in question is basically to look at a set of things and find the commonality. That's an ideal question from a verification perspective: the answer is hard to find but cheap to check.
The specific method outlined here seems pretty doable by today's models, which don't seem smart enough to have some kind of 4d chess hidden alignment.
You just have to go through a ton of all of a model's neurons and identify the associated motif, then use that model as a research model to figure out how to tune out bad behavior.
It's pretty hard to imagine today's models subtly recognizing a motif for bad behavior in the research model and then lying about it. Or lying about the results of an alignment training run. It's possible, I guess, but it seems unlikely, and I don't really think there's any evidence of current models having that variety of misalignment.
I hate to go completely absurdum, but how do you know your CPU or GPU can be trusted?
For one, they are decades old, you can inspect exactly how they flow, are (mostly always) deterministic (given error-correction algorithms) but mostly: the theory behind them is well understood - it had to be before building the hardware. Whereas in ML/"AI" theory is about 20 years behind applied practice - because you can implicitly train something on data and rewards, without exactly knowing how its internal works - leaving interpretablitity and determinism.
1 reply →
Same as how do we know our brains can be trusted. If there are no external voices (scitzophrenia or phoning home respectfully) then the computing device is ours. Hardware bugs are possible but with proven absence of help from the vendor at least the bugs are not malfunctional.
2 replies →
> The vast majority in models up to trillions of parameters are legible, and their circuits beautiful.
That's very encouraging. For quite a while, neural nets were considered an incomprehensible black box. That paper shows they're not. Real progress.
I'm afraid in it's current form they can declare some victory, but may not guarantee it the next month's iteration.
Not sure if there is formal arXiv paper to be released other than this long form Goodfire AI blog post (love the animations!), but if what Nick demonstrated/observed were indeed true, this will resolve the Super intelligence alignment issues that p(doom)-ers have been crying aloud in the past few years.
The most iconic line was the last sentence:
This feels like the watershed moment, again if Nick is proven right, as massive as the AlexNet victory in the 2012 ImageNet Large Scale Visual Recognition Challenge, exactly as Nick pointed out.
Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
It's indeed a big advance for interpretability, but that alone doesn't solve alignment. It doesn't even solve interpretability, as there is still a large number of unexplained neurons, just fewer than with earlier approaches.
Once interpretability is solved and we have a fully human-readable accurate explanation of how a model produces its output, that still leaves the problem of spotting problems ahead of time. E.g. for the hummingbird misclassified as ice cream https://already-clean.goodfire.com/assets/september-revision... the method would have to flag that the texture of ice crystals and of feathers, and the shape of a spoon handle and a bird's beak are insufficiently separated. Purely based on the explanation, without having to rely on trial-and-error to find corner cases like this.
And even when it becomes possible to automatically spot issues, that doesn't mean they will all be solvable. Maybe there are tradeoffs where fixing one thing breaks another, and which choice is the better one depends on information you don't have.
Nick has posted some follow up to beware about over-claiming on this. You still need to interpret the circuits, as well.
Lots of work still to be done, this work just dissolves an early blocker (if it is substantiated).
[dead]
[flagged]
I’m not flagging anything!
What did it say? One comment is totally flagged away now
Nick asked in late November 2024 how he could bet on the country getting better over the next few years. Iirc the suggestion was driven by the idea that merit is going to be taken more seriously now.
Absolutely massive L that 1) he wasn’t already in index funds and had to ask and 2) thought the election of this monster would lead to a better country
Decoupling or not I can’t trust anybody that would ask something so stupid driven by an opinion so easily disproven.
Tide goes in, tide goes out. Im low key of the opinion that these tools enhance existing attributes while displacing the advantages of unearned social or financial power.
So in the short term the tide might go out but then it might tsunami back.
Same goes for this terrifying political feeling around the world. Might simply be a boomer swansong and the reaction might snap back.
> displacing the advantages of unearned social or financial power
A technology that lets you directly buy intelligence with money will displace financial power?
1 reply →
> displacing the advantages of unearned social or financial power
In what way? Sure looks like pay to win to me.
3 replies →
I sincerely hope you are right.
1 reply →
> I can’t trust anybody
I don't know this Nick fellow but i don't think he's asking you to take anything on faith. Either it works out or it doesn't, he's just saying "it's looking good so far".
Newton lost a huge amount of money in the South Seas bubble
Alignment is unsolvable for the very simple reason that LLMs are trained on the net content of human-generated literature, which includes the most Machiavellian deceptive manipulative themes human beings have ever come up with - it all there in the weights.
The solution is to put a dumb machine as an overseer of the smart machine, like in a supermax prison. Not good for profitability, but we all knew this was going in the direction of compilers. How much did you pay for gcc or clang this month?