Comment by lukewarm707
3 days ago
It reminds me of this quote from Judith Butler:
"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."
It always amuses me when people throw a fit when they can't understand technical philosophy written by experts, for experts. You never see someone posting an excerpt from a physics journal and being like "ugh so inaccessible!!!"
Like, I'd get maybe if she wrote this in a NYT opinion piece– but this was written an obscure book you'd have to go out of your way to find.
BTW, this is not that hard to comprehend if you're familiar with continental philosophy, go figure!
Sokal and Bricmont have debunked your amusement eons ago.
The irony is that the experts don't understand it half the time either.
Why? The neural network paper is straightforward and has lots of illustrations showing concrete examples of what happens as images move through the stages of a classifier.
The above quote is from "The world's worst writing".
Let me quote the opening paragraph of the paper:
"After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread. What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
> > "a network’s own mechanisms are its legible building blocks"
Indecipherable.
> > "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
State the claim in plain english. Use the active voice.
"Concepts do not overlap arbitrarily."
> > "What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
Jargon!
"Concepts overlap in a recurring pattern."
It's a research paper targeted at other researchers; you do need to understand some of the field for it to make sense. It's not written to be MechInterp 101.
But it's not bad writing. It's actually very clear. I'm not an professional AI researcher, I just read papers for fun, but I found everything in Nick's article clearly articulated and easy to understand. It's actually one of the more digestable papers I've come across this year.
Here's a translation to easier language:
> "a network’s own mechanisms are its legible building blocks"
To understand this you have to know that up until now attempts to figure out what's going on inside neural networks have relied on indirect techniques, in particular by training another neural network to "unpack" what's happening in the first. This was driven by the belief that a deep neural network isn't directly readable (legible) because the neurons do so many unrelated things simultaneously.
Nick is saying that actually, you don't need a second neural net. You can just read out what's happening directly from the first's own "mechanisms" (arrangements of neurons and weights between them).
> "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
In simpler English: we thought individual neurons did lots of unrelated things, but that's because the standard evidence we had for that belief was misunderstood.
"Concepts do not overlap arbitrarily" and "Concepts overlap in a recurring pattern" don't mean the same thing as what he's saying.
3 replies →
There is indeed a huge amount of useless jargon and IMO deliberate obscurantism in deep learning research.
Sometimes it is to defensively ward off critics, sometimes it is to woo the naive and ignorant, and sometimes it is because the basic ideas are really just incredibly simple and based on simple mathematical intuitions or hunches.
In the latter case, papers and authors appeal to some obscure or overly-complicated technical theory (which usually doesn't actually apply at all to current deep models as they are actually implemented, e.g. infinite-width networks and other such purely theoretical abstractions) to cover up the fact that they really just had a lucky intuitive guess (and likely that twenty other intuitive guesses just didn't pan out). This appeal to theory is often self-deception as much as it is appealing to academic reviewers.
I think this article is a bit of Nick wooing himself and others here... I'll wait for his supposedly forthcoming paper to see if there is any real meat in it.
3 replies →
[flagged]
This is really not that complicated a quote anyway. It is packed full of jargon, but she's talking about moving from a structuralist mindset to one that views power structures as formed contingently from their society rather than structurally from capital.
Not surprising that this philosophical philistinism is from an Anglo publication.
She's a charlatan/fraud, queen of fashionable nonsense. Reject her.
Butlerian Jihad?
Continental philosophers do frequently write complete nonsense (often intentionally), but this isn't hard to follow.
> Althusserian theory
though I think using this term is rather un-feminist because of the part where he killed his wife, but basically continental philosophers will forgive you for anything as long as you're a French communist.
Halte, tu serres ! as another well-known French communist used to call him.