Comment by RGS1811
4 days ago
This piece circles, without naming it, around the concept of “deontic scorekeeping” central to the work of Robert Brandom on semantics. The idea that our ability to create and sustain conceptual systems depends on the give and take of reasons within a social setting. Brandom has been front of mind for me lately because elements of his theory both explain why LLMs should work in principle, and where they are bound to be radically deficient. An LLM encodes the normative regularities of our linguistic/conceptual games into its weights, but it cannot be a real participant in our games because it is incapable of staking durable commitments or changing its dispositions over time.
Worth reading: Robert Brandom, “Making It Explicit: Reasoning, Representing, and Discursive Commitment”
> An LLM ... cannot be a real participant
The participant is not an LLM, the participant is a company. The company learns. It adapts. It has a trillion dollars of skin in the game.
I mean, many companies operate within particular domains but don't have much skin in the domain itself in a way that is meaningful to other participants there -- like Oracle builds software for healthcare, ostensibly, but any actual user of their products will tell you they are nightmarish with regard to usability, and research shows that patients are often harmed as a result. So healthcare isn't their game, it's selling healthcare software.
Likewise OpenAI doesn't have "skin" of any shape or value in e.g. the mathematics community or even in mathematics research per se. In my cynical view, they are exploiting the exposure of mathematical research to automation in an effort to create a case study of their product obsoleting a highly visible and well-leveraged and relatively small profession. Regardless, I doubt even they would say they have durable skin in mathematics in any sense, certainly not in either of the senses I mentioned above. But their output inhabits the same space as mathematicians, who have to contend with the consequences
OpenAI most certainly have a "skin" in math and physics.
They hire math and physics grads and good command of math is very useful in computer science, machine learning and AI.
2 replies →
> The company learns. It adapts
And potentially steals [1].
[1] https://en.wikipedia.org/wiki/Navier%E2%80%93Stokes_priority...
I don't see any theft in that article. Who was deprived of what property? Why didn't they make backups?
2 replies →
> The idea that our ability to create and sustain conceptual systems depends on the give and take of reasons within a social setting.
I read this sentence 5 times and I still cannot parse it.
I think it means if we all have to justify ourselves to each other, we collectively come up with an sustainable ecosystem of ideas that we can understand. So if we stop justifying ourselves to each other, the ecosystem doesn't get built or sustained, and we stop understanding things. Not the person you responded to, but that's my reading.
If a short sentence like that could accurately summarize Robert Brandom’s deontic scorekeeping, he would have wasted a good chunk of his career.
I mean this with zero snark: if you’re interested, you’d be best off asking an LLM or doing a search. But the overall idea is that Brandom described how concepts and meaning are created socially by a kind of game in which people keep track of the claims that other people commit to.
If you accept Brandom’s framework, then it can be argued that LLMs can’t meaningfully participate in this process. This could be considered an explanation of why people have a negative reaction to LLM involvement in activities that are meaningful to humans - even though most people won’t be familiar with the theory, they would instinctively recognize that something important is lacking, such as commitment over time.
What a coincidence, I've also picked up inferentialism but never really got into it.
I'm interested in the reversal that Sellars proposes - it seems clear to me that "representation" is not prior to the space of reasons, despite how unintuitive it is.
> An LLM encodes the normative regularities of our linguistic/conceptual games into its weights, but it cannot be a real participant in our games because it is incapable of staking durable commitments or changing its dispositions over time.
I see no reason why this can't be the case in the future, even though I think inferentialist-based theories are generally strong. Can't you force LLM's (or whatever AI architecture will exist in a decade) to stake durable commitments itself, then? They're at least autonomous enough to run computers. I agree that that is a core part to true meaning & autonomy, but nobody just lets them yet. I don't think inferentialism is a consciousness-based argument, either, so that escape hatch isn't available. I also don't think this is a necessary condition to engage in the space of reasons (but maybe then I'm misunderstanding it, or retreating to a more Sellarsian understanding). Could you explain on this more?
Though again, I only have the book sitting on my desk, and I haven't cracked it open...
I actually agree with you. LLMs could become participants in the future, but this would require (1) durability of real awareness over time (an architecture change from the current fixed-weights + transient context architecture) and (2) that humans grant them the status of equal participants (probably not a terribly high bar given how people already treat these things). I'm not aware of any real progress toward (1), and all the stand-ins people have developed live in the context layer or textual artifacts, which are poor stand-ins for durable memory.
And yeah, consciousness based arguments are bunk IMO. It's clear that LLMs are in a basic sense "aware", and even to some extent "sapient" (in Brandom's usage), i.e. they can justify claims and object to things at least locally. I follow Rorty on the question of consciousness, which is to say that consciousness has no deeper meaning in practice beyond "are you awake", and in philosophical discussion it's largely a proxy for "are you one of us" in some amorphous sense.
Well that's too bad, since Rorty completely misses the point on consciousness.
Consciousness, the interesting part anyway, is qualia, or the "it is like something to be me" as Nagel put it.
Intention, as Dennett used it in his book "The Intentional Stance" is the "are you one of us" angle, and is distinct from consciousness because as the book title suggests, it is a stance taken with respect to some other system (person, cat, dog, turtle, LLM) rather than a description of subjective experience.
So I think it is your (and Rorty's) positions that are bunk. It is very, very far from clear that LLM's are in any basic sense "aware" let alone "sapient". They are extraordinarily able to manipulate language in a way that leads to text generation that has traditionally been assumed to indicate consciousness and/or intelligence. But we have no way of knowing if it is like anything to be an LLM, even though most of us are busy taking the Intentional Stance towards them.
2 replies →
This sounds super interesting but 477 pages is a beast. Would you recommend anything about this subject thats a bit more approachable?
Brandom wrote a shorter version of the book, called "Articulating Reasons", which is available in full on his faculty page [1], or from booksellers. He's a very wordy writer (kind of exhausting to read, even for someone who enjoys philosophy), but if you get ahold of Making It Explicit, chapters 2 and 3 have the meat of the ideas I'm talking about here.
[1] https://sites.pitt.edu/~rbrandom/Courses/2022%20Phil%20of%20...
Edit: One of my ways of benchmarking new AI models is actually to type a version of this prompt in and see how sophisticated the response is: "Talk to me about the concept of deontic scorekeeping in the context of llms". You'll get good mileage out of that.
I think in future, each company will built its own universe of mathematics consisting of prized discoveries. Maybe one Mathematical Universe per Company in the multiverse of mathematics.
Ok what about memory systems though (basically LLM modifies its own RAG context) - that should allow commits and changing dispositions over time?
RAG context memory and modifying the weights are two different things. As far as I know, models can't modify their weights reliably.
Humans change over time due to what they experience. It’s different from waking up from a ten-year coma and having access to a record of what happened to yourself in an alternative timeline where you didn’t fall into the coma.
AI systems have Reinforcement Learning
tbf naming something "deontic" is not helpful
This may not be helpful or strictly accurate, but when I see deontology or variants of that word, I substitute "SHOULD". It is like a duty or obligation.
Super roughly:
* Virtue ethics: character and motivations
* Deontological ethics: rules and obligations
* Consequentialist ethics: utility and outcomes
NOTE: This would suggest a surface reading of Brandom's "deontic scorekeeping" might be keeping score of who has what obligations, who is following or breaking rules, etc.
Then the argument against LLMs being party to that is that we don't really assign roles and responsibilities to LLMs, they don't really have "duties" or "obligations" in the way a member of society will have. We don't negotiate with them over who should do what, they don't contribute to the rules that are set for them.
I'm literally just guessing based on the context here though, since the few times I've tried to watch podcasts with Brandom I generally bounce after 15-20 minutes feeling I haven't really understood anything he is saying.
The irony of complaining about an academic jargon word in a thread about mathematicians.
Mathematicians always define their terms exactly, in order to begin their discussion. Philosophers never define their terms exactly, because then they would have nothing to discuss.
1 reply →
I don't think that's true. Why is it that I see more people using "shaped" and "load bearing"?
Because language is mimetic, and people are consuming huge quantities of AI-generated text. I.e. saying that an LLM can't be an equal participant in our language games isn't saying that it can't play by the rules. In fact that's what LLMs are trained to be best at.
What an LLM can't do (at least with the current architecture) is sustain a commitment that it can be held accountable for later. It doesn't have durable "perspective". It can't update its priors. Maybe a new architecture will come along that gives LLMs memory plasticity, at which point they will shift into being individuated quasi-persons, but that's not what we've got now.
I've written some about this here, if you're interested: https://elliotmilco.substack.com/p/deontic-scorekeeping-comm...
Rather the opposite, AIs are arguably deontically asymmetric because they are powerful and virtual digital entities, and thus should be very concerning to the human species. Not because they current have no internal abstract state for learning state like humans and animals.
You don't consider recursive self-improvement a step towards 'updating a perspective'?
I think you are also suggesting Mathematics and Language are the same thing.
2 replies →
Interesting point
It would only be true if you consider Mathematics and Language to be the same thing.
Both depend on Logic, which is where "deontic scorekeeping" comes in. You can only establish something (score) is true and believable by 'keeping score' and having some level of understanding about the language or math you are reading.
The article is mostly referring to the connection that comes from shared understanding between humans, particularly mathematicians, as they build new shared understanding, and realize there are limits to what they can know and 'leave behind' logic for others to pick up and carry on.
That connection has been severed, or strained a bit, but I'm sure new ones will be created. It just might be of the same flavor that mathematicians prefer, therefore they weep.
And if I understand "deontic scorekeeping" correctly, these forums are a better representation of the concept. You will lend some amount of belief to what is said/read based on what you know.
Ideas are contagious and AI is our most powerful vector yet.
I suppose we would see some interaction with AI in that sense, but it'll take place over the course of learn cycles and when AI starts ingesting its own ideas it typically goes wrong somewhere down the line.